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Preface 


It is my pleasure to present this book on optimal control geared toward chem- 
ical engineers. The application of optimal control is a logical step when it 
comes to pushing the envelopes of unit operations and processes undergoing 
changes with time and space. 

This book is essentially a summary of important concepts I have learned in 
the last 16 years from the classroom, self-study, and research along with in- 
teraction with some great individuals including teachers, authors, peers, and 
students. The goal of this book is to provide a sufficiently detailed treat- 
ment of optimal control that will enable readers to formulate optimal control 
problems and solve them. With this emphasis, the book provides necessary 
mathematical analyses and derivations of important results. It is assumed 
that the reader is at the level of a graduate student. 

Chapter 1 stimulates interest in optimal control by describing various pro- 
cesses and introducing the mathematical description of optimal control prob- 
lems. Against this backdrop, readers are introduced to the basic concepts 
of optimal control in Chapter 2. The notion of optimality is presented and 
analyzed in Chapter 3. The ubiquitous Lagrange multipliers are introduced 
in this chapter. They are elaborated later in Chapter 4 along with important 
theorems and rules of application. Chapter 5 presents the celebrated Pon- 
tryagin's principle of optimal control. With this background, Chapter 6 puts 
together different types of optimal control problems and the necessary condi- 
tions for optimality. Chapter 7 describes important numerical methods and 
computational algorithms in a lucid manner to solve a wide range of optimal 
control problems. Chapter 8 introduces the optimal control of processes that 
are periodic and provides relevant numerical methods and algorithms. A brief 
review of mathematical concepts is provided in Chapter 9. Chapter-end bib- 
liographies contain the cited references as well as important sources on which 
I have relied. 

For an introductory one-semester course, instructors can consider Chap- 
ters 1-3, the main results from Chapters 4 and 5, and Chapters 6 and 7. An 
advanced course may include all chapters with obviously less time devoted to 
the first three. Chapters 7 and 8 may form a part of an advanced optimization 
course. 

Containing all relevant mathematical results and their derivations, the book 
encourages self-study. During initial readings, some readers might want to 
skip a derivation, accept the result temporarily, and focus more on the ap- 
plications. A working knowledge of computer programming is highly recom- 


xiii 


xiv 


mended to solve optimal control problems — whether one intends to write 
one's own programs or use software and programs developed by others. 

Optimal control is the result of tremendous contributions of wonderful 
mathematicians, scientists, and engineers. To list their achievements is a 
formidable task. What I have presented in this book is what I could under- 
stand and have first-hand experience with. I hope the savants will help me in 
improving this book and the students will find the book useful. 

I am profoundly grateful to Dr. Anil Mehrotra, Dr. Ayodeji Jeje, and 
Dr. Robert Heidemann for their assiduous mentoring and training during my 
doctorate and postdoctoral fellowship at the University of Calgary. I am 
thankful to my graduate students, especially Amir Sani, Hameed Muhamad, 
Dinesh Kumar Patel, and Vishalkumar Patel for helping with the proofread- 
ing. I acknowledge Allison Shatkin, Karen Simon, and Marsha Pronin at CRC 
Press who have offered superb assistance in the writing of this book. 

I am very appreciative of the outstanding contributions of the developers of 
TEX, ATEX, MiKTpEX, Xfig, Asymptote, AUCTEX, and GNU Emacs — the 
primary applications I have used to prepare the book. 

Finally, my wife Deepa provided unstinting support and encouragement to 
follow through with this project. My children Jahnavi and Pranav were very 
patient with me all along. 'To the three of them I am deeply indebted. 


Toronto Simant R. Upreti 


Notation 


Vectors 


We will use lower case bold face letters for vectors. For example, 


Yi 
Ya 


Yn 


is a column vector. It has n components. The transpose of y is 


y = E Ya en Un 


where y? is a row vector. Also, y(t) means that each component of y is time 
dependent. 


Function Vectors 


A function vector is a vector of functions. For example, the n-component 
function vectors f(t) and g(y, u) are, respectively, 


ION a.u) | g1(Y1,Y2;---> Yn, Ue ta... sus) 

fa(t) go(y, u) go(Y1,Y2;---> Yn, U1, ise, Um) 
and . Or 

falt) gn(y, U) guis Y2; -- - Uns 141519; -, Um) 


where u is an m-component vector. The function arguments can be functions 
themselves, as in gly(t), u(t)]. 


XV 


xvi 


Matrices 


We will most often use upper case bold face for matrices, e. g., 


11 (12 E Ain 

0421 022 e 2n 
A= 

am1 Gm2 ... Amn 


which is an m x n matrix. The matrix components can be functions. 


Derivatives 


We will use an over dot ` to denote the derivative with respect to time t. A 
prime ' will denote the derivative with respect to an independent variable 
other than time such as z according to the context. Thus, 


a 


: dy / dy " 
ms D = 29 d = 
AE E e 


y! dy 
de dx? 
Partial Derivatives 


Often, we will use the subscript notation for partial derivatives. For example, 


| 0H OH, 0 (aH 


Derivatives Involving Vectors 


T'he derivative of a scalar f with respect to vector y is a vector fy made of 
components that are partial derivatives. Thus, 


xvii 


The derivative of a vector f with respect to a scalar, say, t, is again a vector 
of partial derivatives. For example, 


Of: O fa O fn 7 


h= == — 2. == 
ot Ot Ot 


The derivative of a vector f with respect to another vector u is a Jacobian 
matrix of partial derivatives. Thus, 


oh ah Df 
Qui duo Qu, 
DES T 
£ = Qui Qu» 7. Dum, 
dh Df, A 
Ou, du» "uk Dum, 


States, Costates, and Controls 


In the optimal control problems, we will most often use 

e y to denote the vector of state variables 

e u to denote the vector of controls 

e A to denote the vector of costate variables 
The above vectors will depend upon an independent variable, which will usu- 
ally be time. 


Miscellaneous Symbols 


A few miscellaneous symbols are as follows. 


H Hamiltonian 
J, L, M augmented functionals 
t time 
te final time 
c transformed time in the range 0-1 


T time period 
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Chapter 1 


Introduction 


This chapter introduces optimal control with the help of several exam- 
ples taken from chemical engineering applications. The examples elucidate 
the use of control functions to achieve what is desired in those applications. 
The mathematical underpinnings illustrate the formulation of optimal control 
problems. The examples help build up the notion of objective functionals to 
be optimized using control functions. 


1.1 Definition 


An optimal controlis a function that optimizes the performance of a system 
changing with time, space, or any other independent variable. That function 
is a relation between a selected system input or property and an independent 
variable. The appellation “control” signifies the use of a function to control 
the state of the system and obtain some desired performance. As a subject, 
optimal control is the embodiment of principles that characterize optimal 
controls, and help determine them in what we call optimal control problems. 


Consider a well-mixed batch reactor, shown in Figure 1.1, with chemical 
species A and B reacting to form a product C. The reactivities are depen- 
dent on the reactor temperature, T', which can be changed with time, t. At 


Figure 1.1 A batch reactor operating with / 
temperature as a function of time over a 
certain time duration 


aA + bB => cC 
x(t) 


T(t) 
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any time, however, the temperature is the same or uniform throughout the 
reactor because of perfect mixing. Such a system is described by the mass 
balances of the involved species or the equations of change. They are differ- 
ential equations, which have time as the independent variable in the present 
case. 

An optimal control problem for the batch reactor is to find the temperature 
versus time function, the application of which maximizes the product concen- 
tration at the final time tf. That function is the optimal control T(t) among 
all possible control functions, such as those shown in Figure 1.2. 


ÎN T(?) 


Ta 


final time, t; 


Figure 1.2 Optimal control T(t) and other possible control functions Ta(t)-Ta(t) 


Let us formulate the above problem for the elementary reaction 
aA +bB — cC 
where a, b, and c are the stoichiometric coefficients of the species A, B, and 


C. Denoting their respective concentrations by z, y, and z at any time t, the 
batch reaction process may be described by the following equations of change: 


dr — E " b 

dy E b 

I = —— br? 1.2 
koexp -pp br (t)y (t) (1.2) 
dz E b 

s —— cr? di, 
qe TREP wp ce (ty (t) (1.3) 
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with the initial conditions 
z(0) = Zo, y(0) = yo; and z(0) =0. 


In the above equations, ko is the Arrhenius constant, E is the constant acti- 
vation energy of the reaction, R is the universal gas constant, and T' is the 
absolute temperature dependent on t. The temperature T(t) is a control func- 
tion, which is undetermined. It can be suitably changed to affect the product 
concentration z at the final time tf. The objective is to find the optimal 
control function T(t) that maximizes z(tf), i. e., 


z(te) te 


d 
I = z(t) fo ai 
z(0) 0 
t 
ko | e E “(t)y?(t) dt (1.4) 
= X — pe . 
cro p RT TY 


o 


subject to the satisfaction of Equations (1.1) and (1.2) with the specified 
initial concentrations xg and yo for A and B, respectively. 

There could be other constraints as well. For example, T(t) should never 
exceed a maximum temperature Tmax, and the concentration of A should 
never fall below a threshold level £min. In this case, the inequalities 


T(t) < Tmax (1.5) 
a(t) > Lmin (1.6) 


also need to be satisfied in the time interval [0, tz]. 


Objective Functional 


Observe that I defined by Equation (1.4) is a function of z, y, and T, which 
in turn are functions of t. Thus, symbolically, 


T= Ia), y(t), T(t)| 


A function such as / depending on one or more functions is known as a 
functional. It will be explained fully in Chapter 2. In the present problem, 
I being the objective to be optimized is an objective functional with the 
function T(t) as the optimization parameter. 

Note that T is an undetermined function of t. The functions z(t) and 
y(t) depend implicitly on T(t) through Equations (1.1) and (1.2) for specified 
constants and initial conditions. The evaluation of J for any particular form 
of T(t) [say Ta(t) in Figure 1.2] requires all the function [T,(t)] values over 
the specified time interval [0, tf]. The objective is to find an optimal form of 
T(t) or T(t), which yields the maximum value of 7 simultaneously satisfying 
Equations (1.1) and (1.2) and any other constraints such as Equations (1.5) 
and (1.6). 
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1.2 Optimal Control versus Optimization 


It is easy to perceive from the above example that optimal control involves 
optimization of an objective functional subject to the equations of change in a 
system and additional constraints, if any. Because of this fact, optimal control 
is also known as dynamic or trajectory optimization. 


The salient feature of optimal control is that it uses functions as opti- 
mization parameters. These functions are called control functions or simply 
controls. The routine, static optimization is a special case of optimal control 
using uniform or single-valued controls such as Ta(t) or Ta in Figure 1.2. Had 
we prescribed the invariance of temperature with respect to time in the pre- 
vious example, the problem would have been that of the routine optimization 
with the goal to find the optimal time invariant temperature from all pos- 
sible choices restricted to be time invariant like Ta. Because this restriction 
(or invariance with respect to independent variable) does not exist in optimal 
control, it has a significant advantage over routine optimization. Let us get 
more details. 


Infinite Optimization Parameters 


A control function used in optimal control comprises a number of values, one 
for each value of the independent variable. That number is infinity if at least 
a part of the function is continuous. Thus, in the previous example of the 
batch reactor, the control T(t) is a set of optimization parameters 


Boe cer MEC ves Ms 


T; = T (ti), tı = 0, h= (tii + At) forl<i<n, th = te 


and At; tends to zero as n tends to infinity. Hence, from the standpoint 
of the routine optimization, optimal control is equivalent to multi-parameter 
optimization. With a significantly greater number of parameters available to 
optimize in general, optimal control unlocks a considerably extensive region 
to search for optimal solutions otherwise unobtainable from the routine opti- 
mization. This striking feature, along with remarkable progress in high-speed 
computing, has made optimal control increasingly relevant today to processes 
and products facing tougher market competitions, stricter regulations, and 
thinner profit margins. 
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1.3 Examples of Optimal Control Problems 


To gain further understanding of the applications of optimal control, let us 
study some examples of optimal control problems. In each problem, there is a 
system changing with time or some other independent variable. The system is 
mathematically described or modeled with the help of differential equations. 
At least one such differential equation is needed in order to have an optimal 
control problem. Appearing in the model is a set of undetermined control 
functions, which determines the dependent variables. The set of controls and 
dependent variables in turn determine the objective functional of the optimal 
control problem. 


1.3.1 Batch Distillation 


Figure 1.3 shows a schematic of the batch distillation process for the separation 
of a volatile compound from a binary liquid mixture. It is heated in the bottom 
still to generate vapors, which condense at the top to yield distillate having 
a higher concentration of a the volatile compound. A part of the distillate 
is withdrawn as product while the rest is recycled to the still. An optimal 
control problem is to maximize the production of distillate of a desired purity 
over a fixed time duration by controlling the distillate production rate with 
time (Converse and Gross, 1963). 

Assuming a constant boil-up rate in the still and no liquid hold-up in the 
column, the process can be modeled as 


dm 


<= u(t), m(0) = mo (1.7) 
Eo y), z(0) = zo (18) 


where m is the mass of mixture in the still, t is time, u is the mass of distillate 
product withdrawn per unit time, and x and y are the mass fractions of the 
volatile compound in the still and distillate, respectively. The initial mass mo 
and the mass fraction xo in the still are known. The distillate mass fraction 
is at any time is known from a relation specified as 


y = y(m, x) (1.9) 


Thus, u, m, x, and y are time dependent. The optimal control problem is to 
find the control function u(t) that maximizes the objective functional 


te 


r= Juba 


0 
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condenser 
distillate 


overhead vapors 


u(t) 
> 
recycle product 


binary liquid mixture 


nca cse M Pls CERE cor nig A ERRE MEM RARE RAR OR ERA CADA steam 
m(t), x(t) 


bottom still 


Figure 1.3 Batch distillation process with distillate production rate versus time or 
u(t) as the control function 


subject to Equations (1.7)-(1.9) as well as the purity specification y* for the 
distillate product given by 


or [uu-v)at=0 (1.10) 


o 
ous 
e 
= 
c 
> 
a 
+ 
o 


over a certain time duration tf. 


1.3.2 Plug Flow Reactor 


Pressure plays an important role in reversible gas phase reactions such as 


A 2B 
ka 
where the number of moles of species changes along a reaction path. In the 
above reaction, while lower pressure favors the forward path, higher pressure 
does the opposite. For that reaction carried out isothermally in a plug flow 
reactor (Figure 1.4), it is desired to obtain maximum product (or equivalently 
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minimum reactant) concentration by controlling pressure along the reactor 
length. 


product 
gas out 


gas with 
y = Yo 


plug flow reactor 


Yo 
z=0 z z=L 


Figure 1.4 Gas-phase reaction in a plug flow reactor 


Assuming the ideal gas law to hold, the model for the reactor of uniform 
cross-section is given by 


dy —kiyP(z) , 4ke(yo — y? P (2)? 
TO, 0) = 1.11 
dz ^ 39y-y (2yo — y)? y(0) = yo Een 


where S is the cross-section area, 7 is the residence time of species inside the 
reactor, y is the concentration of the reactant À, z is the independent variable 
denoting the reactor length, P is pressure, and kı and kz are the forward and 
backward reaction rate coefficients (van de Vusse and Voetter, 1961). 

The optimal control problem is to find the control function P(z) that min- 
imizes y at the reactor end z — L subject to Equation (1.11). The minimum 
y is the objective functional given by 


y(L) L 3 
I = y(L) = (0) + / dz = wo + | Paz (1.12) 
y(0) 0 
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1.3.3 Heat Exchanger 


Figure 1.5 shows a single-tube heat exchanger used to heat (or cool) the fluid 
flowing inside the tube by controlling its wall temperature Tw as a function of 
time t (Huang et al., 1969). At any time, Ty is uniform along the z-direction, 
i.e., the length of the heat exchanger. 


tube wall at Ty(t) 


fluid at To fluid out 


heat exchanger 


O = Tut) 


To 
z=0 z zzL 


Figure 1.5 A single-tube heat exchanger 


Corresponding to a given wall temperature 0, it is desired to attain a steady 
state temperature distribution Ts(z) of the fluid in a specified time interval 
[0, te]. For the control to be meaningful, tf is less than the time spent by the 
fluid inside the heat exchanger. A simple heat transfer model of the heat 
exchanger is given by 


OT | dT h 


aa zo UO - T] (1.13) 


where t and z are the independent variables denoting time and heat exchanger 
length, T and Ty are the temperatures of the fluid and the wall, v is the 
average fluid velocity, h is the wall-to-fluid heat transfer coefficient, and Cp 
is the specific heat capacity of the fluid of density p. While T' depends on z 
and t, Tw is a function of t only. The initial and boundary conditions are 


T(z,0) =T(0,t) = To (1.14) 
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where To is the fluid temperature at the inlet of the heat exchanger. 

The steady state temperature T;(z) is the temperature defined by Equa- 
tion (1.13) with the time derivative set to zero and Ty(t) replaced with 0. 
'Thus, 

dT; h 


= pas (1.15) 


Subject to the satisfaction of Equations (1.13)-(1.15), the optimal control 
problem is to find the control function T, (t) that brings in time tr, the final 
unsteady state fluid temperature closest to the steady state wall temperature. 
Hence it is desired to minimize the objective functional 


L 
1= ftrt 2, tt) — Tol dz 
0 


where L is the length of the heat exchanger. It may also be required that 
Tw(t) does not exceed some maximum value, i. e., 


Tw (t) < Tmax 


throughout the time interval [0, te]. 


1.3.4 Gas Diffusion in a Non-Volatile Liquid 


This example shows the application of optimal control to determine a system 
property (diffusivity) as a function of another system property (concentra- 
tion). 

Consider the diffusion of a gas into an underlying layer of a non-volatile 
liquid such as heavy oil or polymer (Upreti and Mehrotra, 2000; Tendulkar 
et al., 2009) inside a closed vessel of uniform cross-section area A (Figure 1.6). 
As the gas penetrates the liquid layer, the pressure inside the vessel goes 
down. The system is at constant temperature throughout the duration tr of 
this process with negligible change in the thickness L of the polymer layer. 
The mass concentration c of gas in the layer at any time t and depth z is 
given by 


Oc & @e c dD(c)  D(c) de ? 
ze enge quip E ja E 1.1 
Ot e T p Oz? a T p de y p Oz (116) 


where D is the concentration-dependent diffusivity of the gas in the liquid, 
and p is its density. Equation (1.16) has the following initial conditions: 
c(0, 0) = csat (t = 0) at the gas-liquid interface (1.17) 
c(z,0) 20 for 0O<zs<L (1.18) 
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9 
SRR 


m" gas 
liquid layer si 
| | fet | | | | [| | | YY ao 
Vo oov o8 o5 o o8 vo voy i 
z= 
Of Of Of Of 
WY: 
diffusion cell 
pressure i 
gas mass in 
liquid 
time 
Figure 1.6 Diffusion of gas into the liquid layer 
and boundary conditions: 
c(0,t) = Csat (t) for 0 « t € tr (1.19) 
a =0 for 0 € t € t (1.20) 
Oz z=L 


where Csat is the equilibrium saturation concentration of the gas at the inter- 
face. 

The objective in this problem is to determine the unknown concentration- 
dependent gas diffusivity such that its use in Equations (1.16)-(1.20) yields 
the calculated mass of gas in the layer 


L 
me(t) = foros. (1.21) 

0 
in agreement with the experimental gas mass me(t). The latter mass is already 
known using system pressure recorded with time and experimental pressure- 


volume-temperature data for the gas. Thus, the optimal control problem is 
to find the control function D(c) that minimizes the objective functional 


te 


I= / [me(t) — me(t)]” dt 


0 
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subject to Equations (1.16)-(1.21). 


1.3.5 Periodic Reactor 


This reactor poses a optimal periodic control problem, which involves periodic 
control functions. Their application can result in better performance relative 
to steady state operation and help achieve difficult performance criteria such 
as those involving molecular weight distribution (MWD). 

Figure 1.7 shows one such application to polymerization carried out in a 
constant volume stirred tank reactor or CSTR fed by liquid streams of initiator 
and monomer. The feed streams react to form polymer, which is discharged 
in the output stream. It is desired to find periodic feed flow rates versus 
time that would produce a polymer of specified polydispersity index (PI), a 
measure of polymer MWD. 

A model for the above process is given by 


dr Fte _ (E, + Fr _ 


e ES z keti (1.22) 
cu = A m D iy — kyxy — kpypo (1.23) 
im —€— € (Fx + Fyne (1.24) 
cla = kay — Cti + kpy(uo + 2m1) (1.26) 


where x and y are the initiator and monomer concentrations in the reactor, 
F(t) and Fy(t) are the undetermined initiator and monomer feed flow rates 
of time period 7, kx and kp are the reaction rate coefficients for initiation and 
polymerization, V is the reactor volume, and o, 1, and pa are the first three 
moments of number distribution of polymer chains (Frontini et al., 1986). The 
subscript f indicates the feed stream property. 

Subject to Equations (1.22)-(1.26), the optimal control problem is to find 
the control functions F(t) and F} (t) that repeat over a given time period 7 
to produce polymer of a specific PI, i.e., minimize the objective functional 


» 2 


_ |1 fotut) ps 
l= fae dt — D 


uit 
0 


where the integrand and D* are the instantaneous and specified Pls, respec- 
tively. The periodicity conditions 


s(t) = s(t- T), s = { Fx, Fy, £, Y, Ho, p1, H2} 
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Figure 1.7 A constant volume stirred tank reactor with periodic feed flow rates 
like those on the right 


must be satisfied simultaneously. 


1.3.6 Nuclear Reactor 


Consider nuclear fission in a reactor where neutrons react with large fissile 
nuclei to produce more neutrons and smaller fissile nuclei called precursors. 
'The latter subsequently absorb more neutrons to produce *delayed" neutrons. 
'The kinetic energy of these products is converted into thermal energy when 
they collide with neighboring atoms. Thus, the power output of the reactor 
depends on the concentration of neutrons available to carry out nuclear fission. 

The power output can be changed according to the demand by inserting (re- 
tracting) a neutron-absorbing control rod into (from) the reactor, as shown in 
Figure 1.8. The control rod upon insertion absorbs neutrons, thereby reduc- 
ing the heat flux and consequently the power output. The opposite happens 
when the rod is retracted. 

The reaction kinetics is given by 


Sw QE Se, NU) sug (1.27) 


aree A y(0) — vo (1.28) 
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inserted rod retracted rod 


reactor core 


control 
rod 
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u(t) = 0 u(t) < 0 


Figure 1.8 Control rod positions in a nuclear reactor 


where z and y are the concentrations of neutrons and precursors, t is time, 
r =rlu(t)] (1.29) 


is the degree of change in neutron multiplication as a known function of control 
rod displacement u(t), a is the reactivity coefficient, 2 is the fraction of delayed 
neutrons, y is the decay constant for precursors, and 7 is the average time 
taken by a neutron to produce a neutron or precursor (Fan, 1966). 

It is desired to change x from zo to a stable value of zr; at time t with 
minimum displacement of the control rod. Thus, the optimal control problem 
is to find the control function u(t) that minimizes the objective functional 


tr 


I= few dt (1.30) 
0 
with the final conditions 
u(t) = x 
d 
I| zi 
dé li; 


subject to Equations (1.27)-(1.29) as well as the constraint 


lu(t)| € Umax 


1.3.7 Vapor Extraction of Heavy Oil 


Vapor extraction or Vapex involves the extraction of heavy oil from a porous 
reservoir (Figure 1.9) using a vaporized solvent close to the dew point. Upon 
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being injected from the top, the solvent drastically reduces the viscosity of oil, 
causing it to drain under gravity and get produced at the bottom. We would 
like to maximize the oil production in Vapex by considering solvent pressure 
versus time as a control function to influence solvent concentration at the top. 


vaporized solvent at P(t) 


injection 
well 


production 
well 


Figure 1.9 Vapor extraction of heavy oil from a reservoir 


Assuming constant temperature and oil density, uniform reservoir porosity 
and permeability, and changes only along the vertical z direction, the solvent 
mass balance is given by 


5; v") (1.31) 


where £ is time, w is the solvent mass fraction in the oil, D is the dispersion 
coefficient of the solvent into the reservoir medium of porosity ¢, and v is the 
downward velocity of the oil. Typically, D is a known function of w. The 
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initial and boundary conditions are 


w[L(0),t] = [PO] (1.33) 


where L is the height of oil in the reservoir with the initial value Lo, wi is 
the solvent mass fraction in the solvent-oil interface at the top, and P is the 
pressure of the solvent. The relationship between w; and P is known a priori 


from experiments. The change in L due to oil drainage is given by 
dL 
dm —v[w(0,t)],  z(0)— Lo (1.34) 


Finally, assuming that Darcy's law holds, we have 


| KK pg 
v= FO (1.35) 


where K, is relative permeability, g is gravity, p is the oil density, and y is oil 
viscosity, which is a known function of w. 

Given the reservoir of length X and thickness Y, the optimal control prob- 
lem is to determine the control function P(t) that maximizes the mass m of 
oil produced at the bottom, i.e., the objective functional 

L(te) 
m=pXY Í dz 
0 


by satisfying Equations (1.31)-(1.35) as well as any other constraint such as 


P< Pax 


1.3.8 Chemotherapy 


The drug concentration y; and the number of immune, healthy, and cancer 
cells (yo, ys, and y4) in an organ at any time t during chemotherapy can be 
expressed as 


dyi 

—— = u(t)— 1.36 
dt u(t) — yey ( ) 
d . _ 

EIS = Vin +72 mem — Y3YaYa — Yaya — o2y» (1 — e VENE (1.37) 
d SE 

ES = rays(1 — Bays) — ysysya — aya (1 — e ^9) (1.38) 
d 

SA = ryya(1 — Biya) — vata — vaya — arya (1 — e) (1.39) 


dt 
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where Y2 in is the constant rate of immune cells that enter the organ to fight 
cancer cells, and u(t) is the rate of drug injection into the organ (de Pillis and 
Radunskaya, 2003). The r;s and f;s are constants in the growth terms, while 
os and A;s are the constants in the decay terms arising due to the action of 
the drug. The 7;s are the constants in the remaining decay terms. Note that 
the drug has toxic side effects since it kills the immune and healthy cells as 
well. 

'To treat cancer using chemotherapy, an objective could be to minimize the 
number of cancer cells in a specified time tf using minimum drug to reduce its 
toxic effects. The optimal control problem in this case is to find the control 
function u(t) that minimizes the objective functional 


te 


E E J u(t) dt (1.40) 


0 


subject to Equations (1.36)-(1.39), and the initial values of drug concentration 
and cell numbers. It is expected that u(t) is never less than zero. 

Other constraints may be present. For example, the number of healthy cells 
during treatment should not fall below a certain minimum, i. e., 


y3(t) 2 Y3,min 
Also, there could be an upper limit to drug dosage, i. e., 


u(t) € Umax 


1.3.9 Medicinal Drug Delivery 


A polymer loaded with medicinal drug provides a means to administer it to 
specific body parts. The initial drug distribution in the polymer matrix could 
be controlled to obtain a desirable pattern of drug release with time (Lu et al., 
1998). 

The one-dimensional release of the drug from the matrix to a tissue in 
contact (Figure 1.10) can be modeled as 


Oc 9 Oc 
— = — D= 1.41 
Ot Oz Ox js] 
where c is the drug concentration, t is the time variable, x is the variable 
along the thickness L of the matrix, and D is drug diffusivity. 
'The initial condition is 
c(z, 0) = u(x) (1.42) 


where u(x) is the undetermined drug distribution. The boundary conditions 


for t > 0 are à 
3 =0 and c(L,t)=0 (1.43) 


x=0 
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Figure 1.10 Drug release from a polymer matrix to a tissue in contact 


At any time, the flux of the drug into the tissue at the point of contact is 
given by 
Oc 
dt ez 


J(t) = (1.44) 


It is desired to match J(t) with a specified drug release versus time relation, 
J*(t) over a certain time duration tr. 


Hence, the optimal control problem is to find the control function u(x) that 
minimizes the objective functional 


te 


I= fve — J* (t) dt 


0 


subject to Equations (1.41)-(1.44). 


1.3.10 Blood Flow and Metabolism 


The peripheral blood flow and metabolism rates versus time in a tissue could 
be non-invasively determined through skin surface temperature, which follows 
the circadian rhythm of sunrise and sunset under resting conditions (Upreti 
and Jeje, 2004). 

Figure 1.11 shows the cross-section of the skin layer around a cylindrical 
limb such as a big toe. 
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skin surface at Te(r,t) 


ambient at T. 


skin tissue with blood 
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Figure 1.11 Cross-section of the skin layer surrounding a cylindrical limb 


The heat transfer model for the skin layer is given by 


oT — k ð „T F(t)Cys(Ty — T) n" AH(t) 
Ot  rpC Or ðr Cp pCp 


(1.45) 


where T is temperature, t and r are time and radial variables, k and C, are 
the thermal conductivity and specific heat capacity of the skin tissue, Cp and 
Ti, are the specific heat capacity and temperature of the blood, and F(t) and 
AH (t) are the undetermined rates of blood flow and metabolism in the tissue. 
The boundary conditions are given by 


ap = h(T — To) (1.46) 
dr r=R 
T(r,0) =T(r,7), ro<r<R (1.47) 


where A is the convective heat transfer coefficient, Too is the ambient temper- 
ature, ro and R are the inner and outer radii of the skin tissue, and 7 is the 
time period of the circadian rhythm. This rhythm enforces Equation (1.47) 
and similar periodicity of F(t) and AH(t), i.e., the equations 


F(0)=F(r) and AH(0) =AH(r) 


Thus, the optimal control problem is to find the two periodic control func- 
tions, F(t) and AH(t), the incorporation of which in Equations (1.45)-(1.47) 
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minimizes the difference between the model-predicted T(R,t) and its experi- 
mental counterpart Te(R,t), i.e., the objective functional 


1.4 Structure of Optimal Control Problems 


The above examples help us identify the structure of optimal control problems. 
As shown in Figure 1.12, an optimal control problem involves one or more 
controls and other inputs to a system under change. The optimal controls 
provide the desired output from the system. 


A set of controls in a system is a combination of 


1. system inputs such as temperature versus time relation in a batch reac- 
tion, 

2. system properties such as diffusivity versus concentration relation in a 
gas-polymer system, and 

3. entities generated in the system, e. g., the rate of heat generation versus 
time relation in a skin tissue. 


open-loop controls 
controls as functions of independent variables 


inputs a system under 
change 
with respect to 
functions relating eee independent 
Ls controls to some ae variables like time 
inputs and space 
feedback 
controls 


optimal control laws 
functions relating 
controls to the state of 
system 


Figure 1.12 Structure of optimal control problems 
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Open-Loop Control 


In most optimal control problems, it is not possible to obtain optimal control 
laws, i.e., optimal controls as explicit functions of system state. Note that 
system state is the set of system properties such as temperature, pressure, and 
concentration. They are subject to change with independent variables like 
time and space. In the absence of an optimal control law, the optimal control 
needs to be determined all over again if the initial system state changes. 

The controls that are not given by optimal control laws are often called 
open-loop controls. They simply are functions of independent variables and 
specific to the initial system state. The application of open-loop controls is 
termed open-loop control, which is the subject matter of this book. 


Closed-Loop Control 


Sometimes, it is possible to derive optimal control laws when the underly- 
ing mathematical models describing the system are simple enough. In many 
problems though, obtaining optimal control laws mostly requires drastic sim- 
plifications of the underlying mathematical models, thereby compromising on 
the accuracy of control. 

Nonetheless, when determined, optimal control laws are easier to implement 
for the control of continuous processes where inputs are susceptible to change 
during long operation periods. Optimal controls are then readily obtained 
from the system state and applied or fed back to the system, as shown in Fig- 
ure 1.12. These controls are called feedback controls and the control strategy 
is termed feedback control, which is a type of closed-loop control. 

A different type of closed-loop control is feedforward control, in which opti- 
mal controls are explicitly obtained in advance from the inputs in conjunction 
with the mathematical model of a system. As shown in the above figure, 
feedforward controls are applied to the system without having to wait for the 
system state the inputs and controls would later generate. 
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Exercises 


1.1 Revise the batch distillation problem in Section 1.3.1 (p. 5) in order to 
minimize the mass fraction of the volatile compound in the bottom still in a 
certain time duration tf. 


1.2 Using temperature T'(z) as a control function at constant pressure, for- 
mulate the problem of the plug flow reactor in Section 1.3.2 (p. 6) to maximize 
the concentration of the intermediate product B in the following sequence of 
elementary first order reactions 


where kı and kz are the reaction rate coefficients dependent on the tempera- 
ture T. 


1.3 Modify the objective functional of the nuclear reactor example in Sec- 
tion 1.3.6 (p. 12) to additionally enable the minimization of oscillations in the 
neutron concentration. 
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1.4 Revise the drug delivery example in Section 1.3.9 (p. 16) to minimize the 
consumption of drug, simultaneously ensuring that its flux does not fall below 
a threshold value over a certain time duration tf. 


1.5 What new conditions would be required if the optimal periodic control 
problem of Section 1.3.5 (p. 11) is changed to a non-periodic one? 


1.6 Compare the use of u(t) in the objective functionals of Equations (1.30) 
and (1.40). 


Chapter 2 


Fundamental Concepts 


This chapter introduces the fundamental concepts of optimal control. Be- 
ginning with a functional and its domain of associated functions, we learn 
about the need for them to be in linear or vector spaces and be quantified 
based on size measures or norms. With this background, we establish the dif- 
ferential of a functional and relax its definition to variation in order to include 
a broad spectrum of functionals. A number of examples are presented to il- 
lustrate how to obtain the variation of an objective functional in an optimal 
control problem. 


2.1 From Function to Functional 


Let us understand the concept of a functional in the light of what we already 
know about a function. A function associates a function value with a set of 
variables, each of which assumes a single value for function evaluation. For 
example, in the equation 


f=a?, a<x<b 


f is a function associating a function value f(x) with the variable x in its 
domain [a,b]. Thus, f(a) is equal to the square of x. This variable assumes a 
single value each time f(a) is evaluated, as shown in Figure 2.1. 

A functional, on the other hand, associates a functional value with a set 
of functions, each of which assumes in its respective domain a set of values 
for functional evaluation. For example, in the equation 


b 
1= f r6): (2.1) 


where f(x) is a continuous function. I is a functional, which associates a 
functional value (f) with the function f in its domain [a,b]. We know that 
value to be the area between the z-axis from a to b and the curve f. Note that 
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Figure 2.1 Evaluation of 
a function f for x = a and 
u=b 


for the evaluation of I(f), the function f assumes not a single value but the 
set of all values from f(a) to f(b). This fact becomes explicit in the discrete 
equivalent of Equation (2.1), i. e., 


n—>00 
3 f lim Az;—0 ( ) 


fi = fin); zı =a; 2=2;1+Azx;, i= 2,3,...,(n—1); and a, =b. 


Note that as the Az; values tend to zero, n tends to infinity. Not a single 
function value but a set of all function values 


{fi, fa, pains fal n — oo} 


is needed to evaluate the functional value I(f). Figure 2.2 shows the eval- 
uation of J for the functions g and h, which are two different forms of the 
function f. 

'To sum up, a function depends on a set of variables, each of which assumes a 
single value for function evaluation. On the other hand, a functional depends 


Figure 2.2 Evaluation of a 
functional J for f = g and 


f-h 
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on a set of functions, each of which assumes a set of values for functional 
evaluation. 


2.1.1 Functional as a Multivariable Function 


A functional dependent on a function is analogous to a multivariable function 
dependent on a vector of variables. For example, consider the functional 


fi” fo 
K = 
(f) f 


dependent on a function f comprising three components fi, f2, and fa. For 
an evaluation of K(f), f assumes a set of values, i.e., a value for each of its 
components. We can say that the components behave as variables. From this 
viewpoint, the functional is equivalent to a multivariable function dependent 
on the variable vector 


T 


f= fi fe h 


To extend the analogy further, consider a functional dependent on an integral 
of a continuous function. The latter has an infinite number of components 
over a non-zero range of integration. Thus, the functional is equivalent to 
a multivariable function dependent on the variable vector comprising those 
components. For example, the functional J in Equation (2.1) is equivalent to 
a multivariable function dependent on a vector of infinite components of f. 

An interesting upshot of the above analogy is that a continuous function is 
equivalent to a vector of infinite components. Thus, f(x) in Equation (2.1) is 
equivalent to the vector 


T 
f = fi fo m În , lim n — oo 


whose infinitely many components are shown in Equation (2.2). 


2.2 Domain of a Functional 


The domain of a functional is a space that holds all possible forms of the 
associated functions or vectors with some common specification. For example, 
the domain of the functional 7 in Equation (2.1) is the space holding all 
forms of f(x) with the common specification that each form be continuous for 
a € x & b. 

We require domains that are linear or vector spaces. Such a space con- 
tains all linear combinations of its elements. The details and the rationale of 
linear spaces are as follows. 
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2.2.1 Linear or Vector Spaces 


In optimal control, we desire to find the minimum or maximum value of a 
functional defined over a specified domain. The analytical procedure is to 
continuously change the associated function from some reference form and 
examine the corresponding change in the functional. The new form of the 
function is, in fact, the result of a linear combination of the reference form 
and some other form of the function in the same domain. This examination 
can continue only if the new form of the function lies within the specified 
domain each time the function is changed. Otherwise, the corresponding 
new functional may not exist or be valid. The validity of the functional is 
ensured by having the specified domain be a linear or vector space. This 
space holds within itself all linear combinations of its elements (functions), 
which are called vectors. A precise definition of linear space is provided in 
Section 9.19 (p. 278). 

In this book, we will deal with functions or vectors that belong to linear or 
vector spaces. Examples of these spaces include a space of vectors of specified 
n components, a space of continuous functions dependent on an independent 
variable varying in a specified interval, etc. 


2.2.2 Norm of a Function 


The aforementioned analytical procedure furthermore requires a single-valued 
measure for the size of the function or vector in the specified domain of a 
functional. That measure is termed the norm. We use it to quantify the 
difference between two functions, or equivalently, the change in the function 
from some reference form. 

The norm of a vector or function y is denoted by ||y||. A norm has the 
following properties: 

1. It is zero for a zero vector or function* and non-zero otherwise. 

2. For any real number a, 

layl] = lalllyll. 
3. The norm obeys the triangle inequality, i.e., 


ly + zll < Hull + Izl 


where z is another vector or function. 


Examples 


In the two-dimensional Cartesian coordinate system, the length of a vector 


y= [u y] is the norm given by 


Iyl = y vi + ya (2.3) 


* [t has all components zero or is zero throughout. 
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For a continuous function f(x) in the x-interval [a, b], a definition of the norm 
could be 


IAI = (2.4) 


We can easily verify that Equations (2.3) and (2.4) satisfy the properties of a 
norm. 

Thus, given a norm, we can compare the sizes of any two vectors. Con- 
sidering a change in a vector or a function, which is reflected in its norm, 
we can study the optimality of the dependent functional by examining how 
it changes. The approach is similar to that in routine optimization where we 
consider a change in a variable and examine the corresponding change in the 
dependent function. 


2.3 Properties of Functionals 


We need three important properties for functionals — continuity, linearity, 
and homogeneity. These properties are required to develop the concept of the 
differential of a functional. 
A functional I is continuous in its domain at a particular function z if for 
every other function y in the neighborhood 
l(y) =1(2) 


lim yz 


A linear functional I over a specified domain satisfies the relation 
Ilay + Bz) = aI (y) + 81(z) (2.5) 


where y and z are any two functions in the domain, and a and £ are any two 
real numbers. A homogeneous functional J of the first degree, on the other 
hand, satisfies the relation 

I(oy) = al (y) 
Note that a linear functional is inherently homogeneous but a homogeneous 
functional is not necessarily linear. The following example illustrates this 
important point. 


Example 2.1 
Consider the following two functionals: 


b b 


y? 
J= ua and k= | ; dx 
yy 


a a 
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where y” denotes the derivative dy/dx. The functional J is linear and therefore 
homogeneous. However, the functional K is homogeneous but not linear. If 
z= ay, then 2 = ay and 

b b 


2,2 2 
Kay) =K() = doma | az akt) 


ay — ay 


a 


showing that K is a homogeneous functional. But K does not satisfy the 
linearity relation, i.e., Equation (2.5), since 
-dr + £ (= 


b 
ara" + ife 
Quo 


K(ay+P82) — RO 


2.4 Differential of a Functional 


The concept of a differential enables the study of the change in a functional 
by rendering it linear and continuous over a sufficiently small change in the 
associated function. The notion follows the definition of the differential of a 
function explained in Section 9.8 (p. 271). 

Consider a change in a functional due to a change in the associated function. 
The differential of the functional is defined as the functional change that is 
a linear and continuous functional depending on the function change that is 
sufficiently small. Thus, given a function change h of size greater than zero but 
less than some positive real number 5, the differential d/ (yo; h) of a functional 
I(y) at y = yo is defined by 


l(yo +h) — I(yo) = dI (yo; h) + e(h) (2.6) 


where the error e(h) vanishes faster than the size of h. In other words, the 
ratio of e(h) to some size measure of h vanishes with the measure. When that 
happens, the error is bound to become zero at some non-zero size of h. Then 
for h of that size or smaller, the change in functional can be conveniently 
represented by the simple functional dI (yo; h) called the differential, which is 
a linear and continuous functional of the function h. 


2.4.1 Fréchet Differential 


The Fréchet differential is defined by Equation (2.6) when the norm of h is 
used for its size. In this case, therefore, the ratio e(h)/||h|| is required to 


Fundamental Concepts 29 


vanish with ||A||. 
Dividing both sides of Equation (2.6) by ||h|| and taking its limit to zero, 
we get the following equivalent definition for the Fréchet differential: 
Tyo +h) = I(yo) — dI(yoh) _ ,, elh) 


= lim —— =0 2.7 
|A]-0 [|^] nio [|All (27) 


Placing no restriction on the form of h, i.e., the shape of the curve h(x), 
this requirement implies that the error should decrease uniformly with h, as 
explained below. 

Let I be a functional of a function y having only two components. Figure 2.3 
graphs the following changes in y from a reference form yo: 


hy, hea, hii, his, haa, and hm,2 
These changes are different forms of h and are shown as arrow vectors. Thus, 


ha is the first change having the norm (or radius) k with a component 
along each of the two mutually perpendicular axes. The norm is given by 


hk2 


Figure 2.3 Changes in a two- 
component function for a Fréchet 
differential 


Equation (2.3). Having the same norm k, hi? is the second change in a 
different direction. The existence of the Fréchet differential at yo requires 
that the ratio e(h)/||h|| should decrease as any h having some norm k or less 
is downsized to any h having a smaller norm. 

For simplicity, let us adhere to two arbitrary directions of h for the same 
norm (i.e., radius), as shown in the figure. Thus, for d/(yo;h) to exist, the 
ratio e(h)/||h|| should decrease as an h having the norm k is downsized in all 
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possible ways — from (i) hķ,ı to hi1, (ii) ha to hi2, (iii) hg» to hi1, and 
(iv) hy o to hia. 

Figure 2.4 illustrates an analogous example with y as a continuous function 
and the norm defined by Equation (2.4). In this case, hz 1(x) and hy (x) are 
two different forms of h(x) of norm k with different directions (Section 9.20, 
p. 279), which are characterized by different shapes of the plotted curves. 

In general, we have the following condition for a Fréchet differential to 
exist. As long as ||^|| decreases, the ratio e(h)/||h||, and consequently the 
error e(h), should decrease uniformly with h, i.e., regardless of its direction. 
If this condition is satisfied, then at some sufficiently small and non-zero norm 
m of h, the error itself would become zero. Then for any h having the norm 
m or less, the functional change is representable by the Fréchet differential, 
which is a linear and continuous functional of h. 


h>0 
nm hga 
M hi 
hi2 
h = 0H — ' A 
. "d A AA 
AP a AS MN a RN A / Nas: 
a x hm, 2 ^ i Fi 
» 
Vei d Am,1 
hy 
x=a x=b 
h<0 


Figure 2.4 Changes in a continuous function for the Fréchet differential 


2.4.2 Gáteaux Differential 


The Fréchet differential is based on the aforementioned condition of uniform 
error disappearance. This condition is too stringent to be satisfied by a large 
class of functionals. If we loosen that condition by preserving the direction 
of h during its downsizing, the resulting differential is known as the Gáteaux 
differential. Denoted by dI(yo; h), it is a linear and continuous functional of 
h defined by 


I(yo + ah) — I(yo) = dI(yo: ah) + €1(ah) (2.8) 
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where the ratio of eı (ah) to a scalar multiplier a vanishes with a itself for 
all h in the domain of I with 0 < ||ah|| < 6. Note that e, (ah)/a is the ratio 
e(ah)/llah|| with the introduction of e. (ah) = e(o)/ ||h||. 
Similar to Equation (2.7), the equivalent definition of the Gáteaux differ- 
ential is 
lim Io +h) — Lo) — H (yo: h) 


a—0 Q 


=0 


which is obtained by dividing Equation (2.8) by a, taking its limit to zero, 
and replacing ah by h in the end. 

Figures 2.5 and 2.6 show the downsizing of h when it is, respectively, a 
two-component vector and a continuous function. Observe the preservation 
of the direction of a vector or the shape of a function during the downsizing 
process. 


Figure 2.5 Changes in a 
two-component function for 
the Gáteaux differential 


When e (ah) becomes zero for a sufficiently small a, then from Equa- 
tion (2.8) 
I(yo + ah) — I(yo) = dI (yo; ah) = ad! (yo; h) 
since d/ (yo; ah) is linear. The above equation can be written in terms of a 


scalar variable 6 as 


dí (yo; h) = tin Luo (oF I~ Myo + BI) ao 


Introducing y = 8 as the reference value undergoing the change (Ay = a) 
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h>0 


hii 


x=a 


h<0 


Figure 2.6 Changes in a continuous function for the Gáteaux differential 


from £ to (B + a), we get 


dI(yo;h) = lim [go + (v + AVA] — Myo + 7h) }y=0 


Avy0 Ay 
d 
m (Yo + yh) 
24 
Replacing y by a, the Gáteaux differential can be written as 
d 
dI (yo; h) = 37 I(yo + ah)a=o (2.9) 


Equation (2.9) delivers a very important result. It shows that the Gáteaux 
differential of a functional is equal to its directional derivative (Section 9.9.1, 
p. 272) along h, i.e., the change in the associated function y at yo. Thus, the 
value of the differential is the derivative of the functional with respect to the 
scalar multiplier to the function change, when evaluated at the zero value of 
the multiplier. 


Example 2.2 
Find the Gáteaux differential of the functional 


1 
»- fo — 2y + 1)dx 
0 


corresponding to the reference y-function yo, and the variation h given, re- 
spectively, by 
yo= 2x and h=x+1 
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Let y = (yo + ah) be a function in the vicinity of yo. Then Equation (2.9), 
along with the above definition of y and specifications of yo and h, provides 


1 
dfi) = Ilyo ahano = 37 fy? = 2y+1 ap do 
y 0 
1 d 1 d 
=|>—y-2+1 jd =] £ y E ain Z 
Ix y —2y t1, dex dy yd i: 
0 0 
1 1 
= | 2 (yo + ah) - 2 , ida — 2 | (yo — Ihde 
0 y 0 


Observe that the Gáteaux differential of J is the functional 
1 
di (yo; h) = 2 fw — 1)hdz 
0 
depending on yo(r) and h(x). As we show below, d/(yo;h) is linear and 


continuous with respect to h, as required by the definition. 


Linearity of the Gáteaux Differential 
The linearity can be easily verified through 
dI (yo; ah + Bk) = ad! (yo; h) + BAL (yo; k) 
Continuity of the Gáteaux Differential 
To establish the continuity, we need to show that 


lim d/ (yo; k) = dI (yo; h) 
kh 


Equivalently, for any function h and a real number e > 0 there exists another 
real number 6 > 0 such that 


dI(yo; k) — dI(yo;h) | < € 
for any function k satisfying 


lh — kl] <ô 
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Let y be the maximum absolute value of (h — k) in the x-interval [0, 1]. 


Then 
1 1 
0 0 


With y related to ||h — k|| as 
y= Rh — kl 


where k is some positive real number, we have 


1. 
O E T | 2 ash [e - 02: 


A == 
Lo 


We can always find a positive real number e greater than the right-hand side 
of the above inequality, i. e., 


2 1h — kl lo < € 


Thus, we can write 
dl (yo; h) — dI(yo;k)| < e 
for which the second-last inequality provides 
Ih—kl| < €/(210) 
N 0 — 


ó 
with the right-hand side standing for a positive real number 5. [] 


The above example introduces an important result. The Gáteaux differen- 
tial of the functional 


is given by 


d dF 
dI(yo;h) = — F = — h 2.1 
(wih) = a fra god Quo 
provided that dF/dy exists and is continuous with respect to y, which is 
defined as y = yo + ah. 
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Example 2.3 
Find the Gáteaux differential of the functional 


1 
= faz 
0 


corresponding to the reference y-function yo and the variation h given by 
yo = 0 and h = 2z, respectively. 

Let y = (yo + oh) be a function in the vicinity of yo. Applying Equa- 
tion (2.10), 


1 


d 
dyrn = J gol he = JE dul rd 
0 a= 


Now at a = 0, we have y = yo = 0, for which the derivative d|y|/dy is not 
defined. The reason is that the right and left-hand derivatives corresponding, 
respectively, to the change in y (i.e., Ay) greater and less than zero are not 
equal as follows: 


ji [yo + Ayl — lyol 
1m ————— E, 
Ay>0 Ay 


e 
| 


yo=0 


Set if Ay c 


m li 
Ayoo Ay 


As a consequence, the Gáteaux differential of J does not exist or is not defined 
at Yo = 0. 


Example 2.4 
In the previous example, if we specify the reference function as 


Yo == +1 


then yo > 0 in the range of the integration. For this specification, both the 
right and left-hand derivatives of |y| with respect to y have Ay > 0 as Ay 
tends to zero. Hence 


and 


[ 


In fact, the Gáteaux differential of J in Example 2.3 is (i) 1 if yo > 0, and 
(11) —1 if yo < 0 in the range of the integration. 
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2.4.3 Variation 


To deal with an even wider class of functionals, the concept of the Gáteaux 
differential is further relaxed to the Gáteaux variation, or simply the variation. 
Similar to the Gâteaux differential, the variation ôI (yo; h) of a functional I 
is 
1. defined by 
Ilyo + ah) — I(yo) = ôI (yo; ah) + ce (ah) (2.11) 


where e, (ah) becomes zero for a sufficiently small a; and 


2. is equal to the directional derivative along h, i. e., 


d 
SI (yo; h) = +I (yo + ah)a=o (2.12) 


However, ôI (yo; h) does not have to be linear and continuous with respect to 
h as opposed to the Gáteaux differential. 


2.4.3.1 Homogeneity of Variation 


Note that ôI (yo; h) is inherently a homogeneous functional of degree one. It 
means that 


SI (yo; Bh) = BSI (yo; h) 


for any real number 8. Recall from Example 2.1 (p.27) that a homogeneous 
functional is not necessarily linear. The homogeneity of the variation arises 
from Equation (2.9) as follows. Let y = a8. Then, from Equation (2.12), 


d d 
M(yo;Bh) = 4--I[yo-ro(8h)ha-o = ggio + Yh)a=o 
a da 
dy d 
= da d; o + Yh)4-o = fól(yo; h) 
Example 2.5 


Find the variation of I(y) in Example 2.2 (p. 32). 
Similar to the Gáteaux differential, the variation is equal to the directional 
derivative 


d 
da To + ah)a=0 


Thus, ôT (yo; h) is the same as the dI (yo; h) in Example 2.2 and is given by 


i 

1 

I(yoih )=2 f(y Dhdz= 5 
0 
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The above variation is of course homogeneous with respect to h since 


1 1 
SI(yo; ah) = 2 [on ahde = 2a | (yo - Dhdx = adl(yo;h) 
0 0 


Generalization to Several Functions 


For a functional dependent on a vector y of several functions in general, the 
above definitions for differentials apply with the vectors yo and h replacing 
yo and h, respectively. For example, the general form of Equation (2.12) is 


d 
SI (yo; h) = 47 I(yo + ah)a=o (2.13) 


da 
Example 2.6 
Find the variation of the functional 


1 T 
Hy) = / (y2-22+1de, y= p(z) yla) 
0 


corresponding to the reference y-function yo, and the variation h given, re- 
spectively, by 


Yo,1 2x hi +1 
= and = 
Yo,2 3c ha x 


Let y = (yo + ah) be a function vector in the vicinity of yo. Then Equa- 
tion (2.13), along with the above definition of y and specifications of yo and 
h, provides 


1 
d d 
dI(yo;h) = 4-I(yo-toh)a-o = a] yi-2y3 +1 , de 
0 
2 — Oye +1 odr = | "aet dx 


2 (Yo 1 + ahı) hj—4 (Yo,2 + aha) ha o=0 dx 
A Mf 


— == 
yı y2 


1 

2 

= 2 | aha — 209 ha) de = -3 
0 
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It can be easily verified that the above variation is homogeneous, i. e., 


91 (yo; ah) = 091 (yo; h) 


2.4.4 Summary of Differentials 


In summary, we have three types of differentials of a functional: 
1. Fréchet differential, 
2. Gáteaux differential, and 
3. Gáteaux variation, or simply variation 


in the order of decreasing strictness for their existence. The Fréchet differential 
requires the error to decrease with function change regardless of its direction 
(or shape). On the other hand, the Gáteaux differential as well as the variation 
require the error to decrease with function change along its direction for all 
possible changes. While both Fréchet and Gáteaux differentials should be 
linear and continuous with respect to a change in function, a variation does 
not need to be so. It is inherently homogeneous with respect to the function 
change. 

In other words, functionals are more likely to have variations than differ- 
entials. Thus, most of the time, we will use the variation of functionals in 
optimal control analysis. Fréchet and Gáteaux differentials will be invoked 
only when their typical properties are needed. 


2.4.5 Relations between Differentials 


The Fréchet differential of a functional is also the Gáteaux differential. In 
turn, the Gáteaux differential of a functional is also the variation. Thus, for a 
functional, the existence of the Fréchet differential implies the existence of the 
Gáteaux differential. In turn, the existence of the Gáteaux differential implies 
the existence of the variation. However, there is no guarantee that the reverse 
relations hold. For example, a functional may have the variation but not the 
Gáteaux differential. Using conditional statements, these relations are 


Fréchet Gáteaux — 
ena] — haa] — 
differential differential 


The following two examples illustrate these relations. 
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Example 2.7 
Consider the functional 


Y1Y2 


I(y) = Yi + Y2 
0 if y=0 


y=0 


where the function (or equivalently the vector) y has two real components, yı 
and y2. Find the variation of I at yo = 0, i.e., for yo,1 = 0 and yo,» = 0. 
In the vicinity of yo, we have 


yi 0 hy ahi 
== + Q —| 

Y2 0 ha aha 

y Yo h ah 


corresponding to which 


ahyh 
Kyo +h) = 1) = 53 


From Equation (2.12), the variation of I is given by 


hiho 
hı + ha 


d 
ól(yo;h) = da Lo + h)a=0 = 
a 
which fulfills the requirement of being first-degree homogeneous, i. e., 


ôI (yo; yh) = yôI (h) 


where y is any real number. This result can be easily verified by replacing A 
by yh in the expression for ôI (yo; h). 


Example 2.8 
In the previous example, does the Gâteaux differential of I exist at yo = 0? 
The Gáteaux differential of J does not exist at yo = 0 since 


d | hih 


J(h) = da 1(W)a=0 ja + m 


is not linear with respect to h. Observe that 
Th) + J(k) = J(h + k) 


where k = ki ko Li is some vector different from h = ha ha T [] 
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2.5 Variation of an Integral Objective Functional 


Let us derive the variation of the integral objective functional 
b 
I(y) — no 


where y and y' (or dy/dx) are functions of the independent variable x. The 

variation has to be obtained in the neighborhood of a reference function yo(x) 

having its derivative y¿(x) with respect to x. If the change from yo(x) to y(x) 
is o h(x), then 

yo(x) + ah(x) (2.14) 

y (a) = vx) + ol) 


From Equation (2.12), the variation of I is given by 


d d 
dI (yo; h) = da Lo + ah)a=0 = ae 


b 
f F(yo + ah, yo + ah’) J 


a=0 
vd 

= / — F(yo + ah, yo + ah’) dx 
a da - 


Using Equations (2.14) and (2.15) in the above equation, 


| > @F dy | OF dy | 
= dx 
a=0 ui =0 


b 
d 
tit) = | da Fly, y) de a Oy da 
b 


f [F (y. y )h + Ey (y, y )h”] J 


a=0 


where Fy and Fy are partial derivatives of the integrand F with respect 
to y and y’, respectively. Expanding y and y’ in the above equation using 
Equations (2.14) and (2.15), and substituting a = 0, we get 


b 
ôI (yo; h) = nim yo)h + Fy (yo; yo) ] de 
It is convenient to express the above variation simply as 
b 
r= firj v uc rs Y da (2.16) 


where it is understood that 
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e y is the reference function yo(x), and y’ is the corresponding derivative 
function y(x); and 
e dy and dy are the functions h(x) and h'(x), respectively. 


In the above equation, the coefficient of dy (i.e., Fy) is called the vari- 
ational derivative of the functional / with respect to y. Similarly, Fy: is 
called the variational derivative of I with respect to y’. If F = F(y), then 
obviously 


b 
I= | Fy(y)dy ax (2.17) 


Example 2.9 
Find the variation of the functional 


T= fo) -ode 
0 


at y = 25 — 1 for dy = —z. 
Using Equation (2.16), the variation of I is given by 


ôI = | [3y?8y — 2y'5y'] dx 


Bd 


When y = z? — 1, y' = 32? so that 


1 
ȘI = [ee — 1)?óy — 6a?6y'] de 
0 


Upon substituting dy = x and the corresponding dy’ = —1 in the above 
equation and integrating, we obtain 
13 
ôI = 1— 
40 


2.5.1 Equivalence to Other Differentials 


Consider the integral functional 


b 
r= | FW) da (2.18) 
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where y is a function of x. If dF/dy exists and is continuous with respect to 
x, then the variation of I is also the Fréchet differential as well as the Gâteaux 
differential. 


2.5.1.1 Equivalence to the Fréchet Differential 
The Fréchet differential d/ (y; h) is defined by 


sim LOY +») — LY) — (ys h) 


h>0 T = (27) 


The above expression is equivalent to the following statement: 
Given an € > 0 there is a ô > 0 such that for ||h|| < 4, 


E is 
[Al] 


Tf the variation ôI (y; h) is equivalent to the Fréchet differential, then the above 
statement must hold for the variation ôI (y; h) substituted for dI(y; h). 
Let us define 
p= | Iy +h) - 1() — ôI (y; h) | 


Then we need to show that p < «|||. 
From the definition of 7 in Equation (2.18) and Equation (2.17), we have 


b 
- l F(y +h) — F(y) —F,(y)h(a) de 


Applying the Mean Value Theorem for derivatives (Section 9.14.1, p. 276) at 
any z in [a,b], we obtain 
F(y+h)-F(y) dF 


—| =F 
y 
for some y in the interval [y — h, y+h]. Substituting this result in the expres- 
sion for p, 
b 


js J E, (9,2) — F,(y,2) h(z) de 


a 
The function F,(y), being a continuous function of y and x in the closed 
interval [a,b], is uniformly continuous therein. Thus, for an e, > 0 there 
exists a ô > 0 such that for ||g — y|| < 4, 


|Fy(g, 2) - Fy(y, x)| <a 


Now in terms of | |, the maximum absolute value of h in the interval [a,b], 
we obviously have B 
Ih(z) < |n] =B8lal| 
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where |A| is defined in terms of a positive real number § and the norm ||h||. 
Multiplying the last two inequalities and integrating with respect to x over 
the interval [a,b], we get 


— ==" 
€ 


b 
[|E 9:9 Paz) n) | de < Bader ll 


where we introduce e = B(b — a)e1. From the triangle inequality for integrals 
(Section 9.22, p. 280), 


b b 
f 69. Etoo ar < f| tea) — Punta) | de 


ÎN = —§ 
p 


The last two inequalities yield 


< ellhl| 


"e | / [F,(9, 2) — Fy (y, he) da 


which was to be proved. 


2.5.1.2 Equivalence to the Gâteaux Differential 


To show this equivalence, we first show that the Fréchet differential of a func- 
tional is also its Gáteaux differential. 

The Fréchet differential, d/(yo; h), is defined by Equation (2.6) if the ratio 
«(h)/||h|| tends to zero with ||h||. Let h = ak where a is a real number and k 
is a function. Then the definition can be written as 
e(ak) 


Ilyo + ak) — I(yo) = dI (yo; ak) + elak), 


im = 
llaxi=0 [Jak 
Since lim||ak|| — 0 is equivalent to lima > 0 for a given k, we obtain 


Ilyo + ak) — I(yo) = dI (yo; ak) + e(ak), lim LM 


=0 


which is the definition of the Gáteaux differential [see Equation (2.8), p. 30]. 
Hence, the Fréchet differential is inherently the Gáteaux differential of a func- 
tional. This result plus the equivalence of the variation to the Fréchet dif- 
ferential of an integral functional, as shown in the last section, proves the 
following: 

The variation of an integral functional is equivalent to the Fréchet as well 
as the Gáteaux differential provided that the derivative of the integrand exists 
and is continuous with respect to the variable of integration. 
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2.5.2 Application to Optimal Control Problems 


In an optimal control problem, we arrive at an integral objective functional 
having the general form 


te 
J= [ uda (2.19) 
0 
where the integrand depends on 


1. the vector of state variables: 


i 
y= yilt) w(t  .. Yht) , 
2. the derivative of y with respect to time t: 
i : : . T 
y= mt) g(t) .. nht), 
3. the vector of controls: 
T 
u= w(t) u(t) — ... =Um(t) , and 


4. the vector of certain undetermined multiplier functions or costates: 


AO Xe n. AQ) 


It is desired to determine the optimal control u that optimizes (i.e., either 
minimizes or maximizes) J. To that end, we need to obtain the variation of 
J to help determine the optimum. The variation of J is a straightforward 
generalization of Equation (2.16) and is given by 


te 


óJ = I x Fy. Syi + 5 f5,09i + 5 Fu du; + 5 nant dt (2.20) 
g e i=1 i=1 i=1 


where the partial derivatives — fy,, fy;, fu;, and fa; — are the variational 
derivatives of the functional J with respect to yi, Yi, ui, and A;, respectively. 
Note that each of dy;, 64;, óu;, and 6A; is a function of t. 

In compact vector notation, 


te 
d= | {iy by + fy by + [but AL 5A} at 
0 
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where dy, dy, du, and ÍA are the variation vectors corresponding to y, y, u, 
and A, respectively. Thus, for example, 


dy = dyi(t) dya(t) es.  OYn(t) 


T 


Example 2.10 
A simpler version of the batch reactor problem (p. 2) for the reaction 


aA — cC 


without any inequality constraints is as follows. 
Find the T(t) that maximizes 


te 


E 
I = ck ———— a^ t 2.21 
0 
subject to the satisfaction of the differential equation constraint 
t = —ak E selt), (0) (2.22) 
= —ako exp ———— x z(0) =a : 
T o EXP RT) > 0 


where & is the derivative of x with respect to t, and zo is the initial concen- 
tration of A. 

To proceed, we need to apply the Lagrange Multiplier Rule, the details of 
which will be provided later in Chapter 4. According to this rule, the above 
constrained problem is equivalent to the problem of finding the control T'(t) 
that maximizes the following augmented functional: 


te 


te 
E E 
J = cko | exp — wats fa —d — akpexp ——À 1” dt 
0 


RT RT 
0 
f E E 
SIL “prt —A &+ ako exp —RT | Pa (2.23) 
€ ———M M — 
f 


subject to x(0) = xo with A as an undetermined multiplier function of t. To 
avoid clutter, we will from now on show the bracketed independent variables 
of functions only if needed for clarity. 

At this point it is sufficient to learn how the augmented functional J is 
formed by adjoining the constraint [Equation (2.22)] to the original functional 
I after bringing all constraint terms to the right-hand side and multiplying 
them by an undetermined multiplier A. 

In the present example, we will find the variation of J. Denoting the inte- 
grand in Equation (2.23) by f and considering the dependence 


f = f(x, 2,T, d) 
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the variation of J is given by 


te 


6J = J (frda + fi6 + froT + f19A) dt 
0 


where fa, fi, fr, and fX are partial derivatives of f with respect to x, i, T, 
and A, respectively. Therefore, 


te 


E ko E 
= | koexp === (c—aA)ax^ ! dq — [Abe + exp — 


RT RT? RT 


0 
a . E a 
(c— ada” óT — t+ ako exp -Rr ? an} dt 
where ôx, ôt, ôT, and 6A are functions of t. The coefficient of 6Ă in the 
above equation is zero because of the differential equation constraint, Equa- 
tion (2.22), which must be satisfied at each t in the interval [0, tf]. Hence, the 
final expression for 6J is 


te 


saa | ko exp -a (c— aA)az?-! da — [doz 


Example 2.11 
Consider the simplified heat exchanger problem (Section 1.3.3, p. 8) in which 
it is desired to find the Tw(t) that minimizes 


subject to the satisfaction of the differential equation constraint 


T = —vT' + zc, «(0 =T (2.24) 


and T(z,0) = T(0,t) = Ty with T and T" as partial derivatives of T with 
respect to t and zx, respectively. The temperature profile T*(x) is specified. 


With the application of the Lagrange Multiplier Rule, this constrained prob- 
lem is equivalent to the minimization of the following augmented objective 
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functional: 
L à L te h 
ral T(x, te) — T* (a) di f [^ Eel AT) dt dx 
—— — A a at 
F G 


0 


(2.25) 
subject to T(z,0) = T(0,t) = To where the undetermined multiplier A is a 
function of the two independent variables x and t. Note the presence of the 
double integral corresponding to these variables. Let us now find the variation 
of J. 
Denoting the first and second integrand of Equation (2.25) by F and G, 
respectively, and considering the dependencies 


F-F[T(zti)] and G = G(T,T',T, Ty, A) 


the variation of J is given by 


L L tg 
ôJ = | eire te) dz + / [ (ei + Gr 6T' + GróT + Gr, 6Tw 
0 0 0 


+ GSA pat dz 


where Fr ,¢,) is the partial derivative of F with respect to T(x, ts), G is the 
partial derivative of G with respect to T', and so on. We have 


. h 
Gr = —T — vT’ + —(T, —T) =0 
à rom ) 


because of the differential equation constraint, Equation (2.24). Thus, the 
variation of J is given by 


L L t; 
51=2/ T (a, te) — T* (x) sra | | A 6T + vóT" + La ôT 
0 0 0 di 
h 
— —- 6Ty| pdtdz 
PCp i 


Example 2.12 
The batch distillation problem of Section 1.3.1 (p. 5) is equivalent to the min- 


imization of 
te 


I= Ju + puly — y*)) dt (2.26) 
0 
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subject to 
m=-u, (2.27) 
t= (sy), (2.28) 


"m 
m(0) = mo, and z(0) = xo with y as a time invariant multiplier used to 
incorporate the purity specification, Equation (1.10), in Equation (2.26). 
The problem at hand has only two constraints, namely, Equations (2.27) 

and (2.28). With the application of the Lagrange Multiplier Rule, this prob- 
lem is equivalent to the minimization of the following augmented objective 
functional: 

te 

=| u + puly — y) + M(—m — u) + Ag -t+ —(e-y) dt 
0 _x _—_ AA 


f 


subject to m(0) = mo and z(0) = xo. Let us now find the variation of J. 
We denote the integrand of the above equation by f and observe that 
f = f(m, m, £, £, Y, U, H, Ai, À2) 


where all arguments of f except u are functions of t. Then the variation of J 
is given by 


te 
óJ -| fmom + fndmt frdxu + fede + fydy + fudut fun + fr, 6r1 
0 


+ fy,dAq dt 


The last three terms of the above equation are zero since 
te 


te 
| fesuat ón [uly v)at=o, fy, =-m-—u=0, and 
0 0 


. u 
ha — i IL(r-y)-0 
m 


respectively, due to the constraints — Equations (1.10), (2.27), and (2.28) 
— which have to be satisfied by necessity. In general, the coefficient of the 
variation of a multiplier will always be zero due to the presence of the corre- 
sponding constraint. Hence, the variation of J is given by 

te 


A 
az= | Aa g) ôm — [Ay ]óm + aa da — [Ag]ót + u Hec] du 
m m m 


+ 1c u(y— y)-AXd Ba y) vu dt (2.29) 


[ 
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Further Simplification 


The variations of integral objective functionals can be further simplified at 
their optima by integrating by parts the terms involving derivative functions 
such as ó$ and dm in Equation (2.29). As a matter of fact, this simplification 
is an important step in deriving the optimality conditions. Therefore, it is very 
important to familiarize oneself with the following formula of integration by 
parts: 


(2.30) 


where y and z are continuous functions of z, with y being additionally differ- 
entiable. Note that the right-hand side of the above equation carries only the 
integral of the second function z — a simplification in case z is a derivative 
function like dz. 


Example 2.13 
For the batch distillation problem, let us simplify 6J given by Equation (2.29) 
when J is optimal. We can get rid of dm and óx in the following two terms 
of óJ: 

te te 

- alărinae and - oaze 

0 0 
using integration by parts. It is possible since the multiplier functions — Aa 
and A2 — are differentiable whenever J is optimal. We will prove the dif- 
ferentiability of multiplier functions later in Chapter 3 on p.78. Note that 


the derivative function has to be the second function z when utilizing Equa- 
tion (2.30). Its application yields 


te te t 


t f dA t 
- [ona - x f sat ZEE 
0 0 dt Jo 


0 0 


te 
te ; 
=- dim + f Asmat 
0 


and similarly 
te te 
t ` 
- | Delăzat = = Dad + | Azozar 
0 0 
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Upon substituting the last two equations into Equation (2.29), we get 


= | NU y) óm4 ba aud ee oy 
m m m 


A 
+ 1+u(y= y") - x + T (e - y) buy de 
te te 
=> Aóm d A»óc 
0 0 


The variation of J thus obtained is readily amenable to further analysis for 
the determination of optimal control. 


2.6 Second Variation 


Let 91 (yo; h) exist at yo as well as (yo + ah) for sufficiently small a. Then the 
variation of ôI (yo; h) at yo is the second variation of the functional J and is 
denoted by ó?I(yo; h). Using Equation (2.12), we obtain 


I h; ah) — ôI (yo; ah 
Hi) = Esrtgosatiaco = luo “Wot hi at) duo) 
da a—0 a 
d d 
=! (yo + ah + Bh)g=0 — =I (yo + Bh)g-o 
xod dg 
= lim 
a—0 a 


With y = B, and Ay =a, we get 


d d 
—T yo + (y + Ay)hly=0 — =I (yo + Yh)y=0 


2 DEN OY dy 
5 I(yoh)— lim, Ay 
d d d? 
= ay wy tdo = gali + Tha 


Replacing y by o, we obtain the second variation of I at yo as 


2 


d 
8^ [(yo; h) = qa (vo + ah)a=0 (2.31) 


2.6.1 Second Degree Homogeneity 


We now show that the second variation is homogeneous of the second degree 
in h, i.e., 


5”I(yo; Bh) = 8?8? I (yo; h) 
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Replacing h by 8h in Equation (2.31), we get 


2 


d 
? I (yo; Bh) = Faz! (vo t aBh)a=o 
a 


Let y = af. Then using the chain rule of differentiation, 


dy dy 


? I (yo; Bh) — = da da 


A, 
Pa 


dy 
=P 51 (yo + yh)y=0 = B751(yo;h) 


2.6.2 Contribution to Functional Change 


The functional change, I(yo + h) — I(yo), can be expressed in terms of the 
second variation provided (d?/da?)I(yo+ah) exists and is continuous at a = 0 
for all h. Using the second order Taylor expansion (Appendix 2.A, p. 52) along 
a function g, 


d? o? 
I(yo + ag) — I(yo) = ôI (yo; g)o + agz T (o + B9)8-«o E rai 


d? d? 
= ól(yo; g)a + agz To + AD) poa daz 1 (yo + ag)a=0 
RM ——4Ó 
o2 €2(ag) 
tg = I< 


where we have subtracted and added 6?/ (yo; g), and used Equation (2.31) for 
the subtracted term. Note that the term ¢2(ag) shown above tends to zero 
with a due to the continuity of (d?/da?)I(yo +ag) with respect to a at a = 0. 

Since 6I(yo;g) and 9?1(yo; g) are homogeneous of the respective first and 
second degrees, 


2 


1 a 
I(yo + ag) — I(yo) = SI (yo; ag) + 39 1o; ag) + ex(o9) > 
With h = ag, and e» = £20? /2, we obtain 
1 
I(yo +h) — I(yo) = ôI (yo; h) + 39 I (o; h) + €2(h) (2.32) 


where the error €2/a? tends to zero with o?. The error is zero for sufficiently 
small a, or equivalently the change along a given A. 
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Generalization to Several Functions 


The above result can be generalized to a functional dependent on several 
functions. Thus, 


I(yo +h) - Iyo) = ôI(yo;h) + 58% I(yosh) +e2(h) (2.33) 


where / depends on the function vector y given by 


ute] [yato)] hi) 
y2(x) yo,2(x) h2(x) 
= + a 
yn (x) Vo,n(x) hy (1) 
inp acd SETS 


and e/o? tends to zero with o?. The error is zero for sufficiently small a, or 
equivalently the change along h. 

We end this chapter by pointing out that the variation of an objective 
functional will provide us with important clues about its optimum, similar 
to what a differential does for an objective function. The second variation 
will provide some auxiliary conditions and help in the search for optimal 
solutions. The necessary and sufficient conditions for the optimum of an 
objective functional will be the topic of the next chapter. As expected, those 
conditions will use the concepts we have developed here. 


2.A Second-Order Taylor Expansion 


Considering I(yo + ah) to be a function of a, the Fundamental Theorem of 
Calculus (Section 9.13, p. 275) gives 


a 


Kyo + ah) — Ig) = / T (yo + Bh) dB 
0 


where ^ denotes the derivative with respect to 8. In terms of 


v=l'(yot+ Bh) and w=(a-— ) 
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we can write 


I(yo + ah) — I(yo) 


| 
S—_ o 
a 
e 
o 
+ 
Xo 
= 
x 
= 
N 
a. 
Uo 
Il 
—, 
e 
E, 
e. 
Co 


a 
B=a 
= vw — few d6 (upon integration by parts) 
B=0 
0 


Q 


o UT REO / [yo + 8h))(e — 8) 48 (2.34) 
5I(yo;h) 0 
Ra 


where 1” (yo + Bh) is the second derivative of I(yo + Bh) with respect to f. 
If m and M are the respective minimum and maximum values of I” (yo + 8h) 
in the interval 0 € 6 <a, then 


m f(a- a8 < Ra <M fle- pă 
0 0 


o? a? 
— < Ra <M— 
or "a I 2 & 7 
Ra 
or m Soga SM 


Hence R2/(a?/2) is bounded by the minimum and maximum values of the 
second derivative, I” (yo + Gh), which is continuous. Therefore, by the Inter- 
mediate Value Theorem (Section 9.15, p. 276), R2/(a?/2) must be equal to 
I" (yo + Bh) for some B in the interval 0 < 8 <a. 

The interval is —o < PB < 0 if a is negative. Hence the combined result for 
a non-zero a is 


2 
a 
Ra = "(yo Bh) P< lal 
Let y = B/a. Then 
B=ay<|al or «1 


and Equation (2.34) can be finally written as [compare with Equation (9.1) 
in Section 9.6 (p. 270)] 


o? d? 
I(yo + ah) — I(yo) = aóI(yo; h) + Faget (wo + Bh)gaya, ly <1 
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Exercises 


2.1 Given two functionals I[y(1)] and J[y(x)], find the variation of 
a. I[y(z)]J[y(z)], and 


CIA 
Jly(x)] 
2.2 In the plug flow reactor problem of Section 1.3.2 (p. 6), substitute Equa- 


tion (1.11) into Equation (1.12) and find the variation of the resulting objec- 
tive functional. 


ere J[y(z)] =0 


2.3 Derive the augmented functional for the optimal periodic control problem 
in Section 1.3.5 (p. 11). Find the variation of that functional. 


2.4 Repeat the previous exercise for the nuclear reactor problem in Sec- 
tion 1.3.6 (p. 12) ignoring the inequality constraints. 


2.5 Simplify the variations obtained in the last two exercises using integration 
by parts. 


2.6 Follow the approach of Example 2.11 (p.46) to derive the augmented 
functional for the blood flow and metabolism problem in Section 1.3.10 (p. 17). 
Find the variation of that functional. 


2.7 Repeat the previous exercise for the medicinal drug delivery problem in 
Section 1.3.9 (p. 16). 
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2.8 The objective functional of an optimal control problem depends on func- 
tions as well as their derivatives. Under what circumstances would the varia- 
tion of such a functional become the differential? 


2.9 Show that the Gáteaux differential of the functional in Example 2.3 
(p. 35) is (i) —1 if yo < 0, and (ii) 1 if yo > 1 in the range of the integration. 


Taylor & Francis 
Taylor & Francis Group 


http://taylorandfrancis.com 


Chapter 3 


Optimality in Optimal Control 
Problems 


This chapter presents the conditions related to the optimality of a func- 
tional. We derive the necessary conditions for optimal control to exist and 
apply them to optimal control problems. We also present sufficient conditions 
assuring the optimum under certain conditions. Readers are encouraged to 
review the logic of the conditional statement in Section 9.25 (p. 282). 


3.1 Necessary Condition for Optimality 


Let I be a functional of a function y. If J is minimum at $9, then the values of 
I for all other admissible functions in the vicinity of cannot be lower than 
the value of I at Y. Precisely, 


I(y) — I(9) 20 


is the necessary condition for the minimum of J where y is any admissible 
function in the vicinity and the norm of the function change (y — $) is less 
than some positive number. Denoting (y — 9) by dy in the above inequality, 


we have 
I(g + ady) — I) > 0 (3.1) 


— — 
y 
where a is a non-zero scalar variable with absolute value less than or equal to 
unity. In other words, —1 < «a < 0 and 0 < a < are the two intervals of a. 
Now if I has a variation at y, then from the definition of variation [Equa- 
tion (2.11), p. 36] and its homogeneity property 


I(à + ady) — I(i) = 01 (3; ady) = aóI(g;óy) = aóI (3.2) 


for sufficiently small a in its prescribed intervals. Hence, from Inequality (3.1) 
and Equation (3.2) 
a6l > 0 


According to the above inequality, 67 should be either 


97 
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1. greater than or equal to zero when a is greater than zero, or 
2. less than or equal to zero when a is less than zero 


Thus, ôI = 0 is the only non-contradicting condition that is applicable for the 
minimum of /. 

In fact, we obtain the same condition if / happens to be maximum at y. In 
this case, we proceed with the observation that I(y) — 1(9) € 0 and do the 
same analysis as above involving a. 


As a consequence, 
(3.3) 


is the necessary condition for the optimum (either minimum or maximum) 
of the functional J. Not guaranteeing an optimum of J, Equation (3.3) is just 
a logical consequence of J being an optimum. 

We will now apply the above necessary condition to optimal control prob- 
lems and derive equations to help identify optimal solutions. 


3.2 Application to Simplest Optimal Control Problem 


Consider the simplest optimal control problem in which it is desired to find a 
continuous control function u(t) that optimizes the objective functional 


I= | F(y,u)dt (3.4) 
| 


subject to the constraint of the differential equation 


Y = 9(y,u) or G(y,y,u)=-—y+yg(y,u) =0 (3.5) 


with the initial condition 

y(0) = Yo (3.6) 
Equation (3.5) is called the state equation since it describes the state of the 
system through the state variable y as a function of the independent vari- 
able t. It is assumed that F and g have continuous partial derivatives with 
respect to y and u. Note that this problem is autonomous in the sense that 
the independent variable does not appear explicitly in F or g. When it does, 
the problem is easily convertible to the autonomous form (see Exercise 3.4). 


Effect of Control 


Observe that the objective functional J in Equation (3.4) is influenced by 
the control u directly as well as indirectly. While the direct influence is 
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through the integrand F, the indirect influence stems from y being affected 
by u through the state equation constraint, i.e., Equation (3.5). 

Hence, to solve the problem, we need to first obtain an explicit solution 
y = y(u) and then substitute it in the expression of F. However, such solutions 
do not exist for most optimal control problems, which are typically constrained 
by highly non-linear state equations. 

The recourse is to adjoin the constraints of an optimal control problem to 
the objective functional using new variables called Lagrange multipliers. Their 
introduction obviates the need to obtain explicit solutions of state variables 
in terms of controls. The optimization of the resulting augmented functional 
is then equivalent to the constrained optimization of the original functional. 
This outcome — which we shall call the Lagrange Multiplier Rule — springs 
from the Lagrange Multiplier Theorem, the details of which will be presented 
in the next chapter. For now we accept this result and work with the aug- 
mented functional. 


3.2.1 Augmented Functional 


In the present problem defined by Equation (3.4)-(3.6), we adjoin the state 
equation constraint to J using a Lagrange multiplier A and obtain the aug- 


mented functional : 
f 


de J [F (y, u) + AG(y, 9, u)] de (3.7) 
“ES > 
0 f 

For the given initial condition, i.e., Equation (3.6), the optimization of J is 
then equivalent to the optimization of I constrained by the state equation. 
Observe how the integrand f in Equation (3.7) is formed by multiplying A by 
G, which consists of all terms of the state equation constraint moved to the 
right-hand side. We will follow this approach, which will later on enable us 
to introduce a useful mnemonic function called the Hamiltonian. 

The Lagrange multiplier is also known as the adjoint or costate vari- 
able. It is an undetermined function of an independent variable, which is t 
in the present problem. Both A and the optimal u are determined by the nec- 
essary condition for the optimum of J. The subsequent analysis expands the 
necessary condition, which is terse as such, into a set of workable equations 
or necessary conditions to be satisfied at the optimum. 


3.2.2 Optimal Control Analysis 


From Equation (3.3), if J is optimum, then 6J = 0. This result, on the basis 
of Section 2.5.2 (p. 44), can be expressed as 


te 
his J (&F +6G -- GóA)dt = 0 (3.8) 
0 
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ôF = Fy0y + F, du (3.9) 
and 


dG = G,dy + Gydy + Gydu (3.10) 
We rewrite Equation (3.8) as 
te te 
iJ = [or +G) + | Gorat=o 
0 


0 
Ne ==“ Noa. — i 
Jı (5y,5u) Ja (2) 


where the first integral (J1) depends on arbitrary functions dy and du, and the 
second integral (J2) depends on another arbitrary function 94. The function 
dy is not arbitrary but depends on dy. 

Note that Jı is independent of 0A while Jp is independent of dy and du. 
Because of this fact, both Jı and Jz must be individually zero. Otherwise, 
Ji (dy, ðu) = —J2(9A), implying the contradiction that Jı depends on YA and 
J2 depends on dy and du. Therefore, Jı = 0 and Jz = 0 must be satisfied in 
order to ensure óJ = 0. 


Satisfaction of Jz = 0 


We consider the equation 
tr 
Ja = T GóAdt = 0 
0 


Since 6X is arbitrary in the above equation, Jz = 0 is satisfied by having 


(3.11) 


throughout the interval [0,tf]. Keep in mind that we have considered the 
optimum of J without supposing that G is zero, as given by Equation (3.5). 
The above equation, G = 0, comes forth as a necessary condition for the 
optimum. 


Satisfaction of Jı =0 


Next, we consider 


te 


(SF + 5G) dt = 0 (3.12) 
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Arising from 6J = 0, both Equations (3.11) and (3.12) are the necessary 
conditions for the optimum of J. On the basis of the Lagrange Multiplier 
Rule, these two equations are also the necessary conditions for the constrained 
optimum of J. Both optima are still subject to the given initial condition, i. e., 
Equation (3.6). The equivalence between the two optima will be shown later 
in Section 4.3.3. 

We now proceed to obtain workable equations that will ensure the satis- 
faction of Equation (3.12), and consequently, be necessary along with Equa- 
tion (3.11) for the optimum of I. 

With the help of Equations (3.9) and (3.10), Equation (3.12) can be ex- 
pressed as 


te te 


n=} EXC, iy NO ză de a ] 6 60a - 0 


where the first integral (J3) depends on the arbitrary function dy, while the 
second integral (J4) depends on another arbitrary function du. 

Note that J3 does not depend on du, and J4 does not depend on dy. There- 
fore, for the above equation to hold, both integrals must be individually zero. 
Otherwise, J3(9y) = —Ja(6u), implying that J3 depends on du and J4 depends 
on dy, which is contradictory. Thus, 


te 


J3 = J (Fy + AG y)Oy + AG OY dt = 0 (3.13) 
0 
and 
te 
a= I (F, + AG, Jóu dt = 0 (3.14) 
0 


must be satisfied in order to ensure J; = 0. 


Satisfaction of J = 0 


For Equation (3.14) to hold for any arbitrary du, 


E PACA =, Oct he (3.15) 


The above equation is known as the stationarity condition. 
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Satisfaction of J4 = 0 


Upon substituting Gy = —1, which is obtained from the definition of G in 
Equation (3.5), Equation (3.13) becomes 


ir te 


i fe +AG,Jóy dt — p» dt =0 (3.16) 
0 0 


Before simplifying the above equation further, we need to express the second 
integral in terms of dy. Applying integration by parts to the second integral, 
we get 


te 


te 
fe, -AG,)óydt — Ady + My + f doy at = 0 (3.17) 
0 “~~ = 0 
The above step assumes that we can differentiate A with respect to t in Equa- 
tion (3.16). It is indeed true, as shown in Appendix 3.A (p. 78). 
Because of the initial condition, i.e., Equation (3.6), there cannot be any 
variation in y at t = 0 where y is fixed. Thus, the third term in Equation (3.17) 


is zero. Rearranging the remaining terms of the equation, we get 


te 


f(r +AGy +A)óydt— Xy =0 (3.18) 
t=t¢ 
0 


From the Mean Value Theorem for integrals (Section 9.14.2, p. 276), 


te 


I (F, +46, +A) öy dt = plts)ou(tr)(te— 0) 
0 wee ë 
p(t) 
where p(t) is the coefficient of dy and 0 € tı € tf. The above equation shows 
that the integral depends on an arbitrary function value óy(t1). We contend 
that the above integral does not depend on dy(ts) though. In other words, the 
above equation is valid for tı less than tf. To prove it, let ti be equal to tf in 
the equation and p(t) be given over the interval [0, tf]. Then, from the above 
equation, 
te 
| nat = tensos (3.19) 
0 
and the integrand pdy 


e either stays constant so that it has the same value p(t,)dy(t¢) at any ti 
less than t; (Case 1) 
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or changes with t, in which case póy attains at least once the same value 
p(te)Sy(te) at tı less than tr (Case 2). 


Suppose this conclusion is not true, so that 
p(ti)óy(t1) > p(te)dy(ts), O< ti «tr 


Then p(ts)dy(t) is the minimum integrand value. Thus, 


te 
f poy dt > p(to)Sy(te)te 
0 
which contradicts Equation (3.19). The other situation when 


p(t1)0y(t1) < p(tr)óy(tr), 0t «tr 


is similarly contradictory. Hence, póy = p(tr)óy(tr) at least once for 
ti «bg 
In either case, 
te 
J init = oot - o 
0 
where tı is less than tf. Thus, the integral is relieved from dependence on 
óy(tt), which therefore affects only the last term of Equation (3.18). As rea- 


soned in similar instances previously, the two terms of Equation (3.18) must 
therefore be individually zero. Hence 


te 
J Fy +AG +À óydt — 0 (3.20) 
0 


and 


A(te)oy (te) = 0 (3.21) 


Now Equation (3.20) is certainly satisfied if the coefficient of dy is zero, or 
equivalently 


N= —-(Fy+AGy), OSt< ts (3.22) 


The above equation is known as the Euler-Lagrange equation in honor of 
Swiss mathematician Leonard Euler (1707-1783) and French mathematician 
Joseph Luis de Lagrange (1736-1813). The Euler-Lagrange equation is also 
called the adjoint or costate equation since it defines the adjoint or costate 
variable A. 

A question arises whether the coefficient of dy in Equation (3.20) really 
needs to vanish at t = 0 where dy is already zero. The answer is yes and the 
proof is provided in Appendix 3.B (p. 81). 
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Table 3.1 Necessary conditions for the optimum of J in the problem defined by 
Equations (3.4)-(3.6) 


. The state equation 


with initial condition 


. The costate equation 


with final condition 


. The stationarity u 


condition 


Finally, we are left with Equation (3.21), which is satisfied if the coefficient 
of the arbitrary dy(t¢) is zero, i. e., 


(523) 


The above equation is in fact the initial condition for the backward integration 
(from t = tf to 0) of the costate equation, i.e., Equation (3.22). 


Summary of Necessary Conditions 


Table 3.1 collects the equations we have used to satisfy 6J = 0. These equa- 
tions are the necessary conditions for the optimum of J or, equivalently, of I 
subject to the state equation constraint. The solution of the necessary condi- 
tions provides the u and the corresponding y as well as the A at an optimum 
of I. Put differently, for I to be optimal, u must be such that these conditions 
are satisfied. Otherwise J cannot be optimal. 


Keep in mind that the satisfaction of these equations does not guarantee 
the optimum but provides a candidate for the optimum. For example, several 
optima may exist and the optimum provided by the necessary conditions may 
not be the best among all optima. (We will explore this issue further in 
Section 3.3.1 later on.) 


Most important, if the equations are not satisfied, then the optimum can- 
not exist. The above facts conform to the logic of a conditional statement 
explained in Section 9.25 (p.282). For now, let us apply these conditions to 
optimal control problems. 
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Example 3.1 
Find the necessary conditions for the minimum of 


te 


T= [eè - mat 


0 


subject to the satisfaction of the differential equation constraint 


i=-VG+u (3.24) 


where u(t) is the control and y(t) is the state variable with the initial condition 
y(0) =0. 
The augmented functional for this problem is 


te 


J= u? — yu +A =Y yy +u pa 
— — — — 
0 F g 
M ë 
G 


with the functions F, g, and G as indicated in the above equation. From 
'Table 3.1, the necessary conditions for the minimum of / are 


1. the state equation, Equation (3.24), with the initial condition y(0) — 0; 


2. the costate equation, À — —(Fy + AGy), i.e., 


ee = 


29 
with the final condition A(tf) = 0; and 


3. the stationarity condition Fu + AG, = 0, i.e., 
2u— y -A-0 


throughout the time interval [0, te]. [] 


Example 3.2 
Find the necessary conditions for the maximum of 
te 
E 
I = k e dt k = ko — — 
J ckz" dt, exP — Bp 
0 


subject to the satisfaction of the differential equation constraint 


t = —akz^ (3.25) 
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with T(t) as the control and z(0) = zo as the initial condition. 
This is the simple batch reactor control problem (see Example 2.10, p. 45) 

for which the augmented functional is 

te 

=| ckz? + A —d — aka" pa 
~ e — o_ 
0 F g 
j| ———— 


G 


with the functions F, g, and G as indicated above. Referring to Table 3.1, T' 
is the control u as a function of the independent variable t and x is the state 
variable y. Using the table, the necessary conditions for the maximum of I 
are 


1. the state equation, Equation (3.25), with the initial condition x(0) = xo; 
2. the costate equation, À = —(F + AG,), i.e., 
À = —aka?-! (c — aA) (3.26) 
with the final condition A(tf) = 0; and 
3. the stationarity condition Fr + AGr = 0, i.e., 


kE 


RT? z^(c— aA) =0 (3.27) 


over the time interval [0, te]. 


Hamiltonian 


For the optimal control problem defined by Equations (3.4)-(3.6), a mnemonic 
to remember the necessary conditions for the minimum is the Hamiltonian 
function defined as 


H = F-+ Ag 
It is easy to verify that 
dH 
— =H = y (3.28) provides y = gly, u) (3.5) 
dà == “== 


state equation 


provides A=-—(F, + AG) (3.22) 
A 


costate equation 


and 
dH : 
—— zH, = 0 (3.30) provides F,4AG,-0 (3.15) 
U — mm” 


stationarity condition 
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Example 3.3 
Repeat the previous example utilizing the Hamiltonian. 
The Hamiltonian is given by 


H =ckx" + A(—akz") 
Therefore, the necessary conditions for the maximum of / are 


1. the state equation, Equation (6.10), obtained from $ = Hj and the 
initial condition (0) = 0; 


2. the costate equation, Equation (3.26), obtained from A = —H, and the 
final condition A(tf) = 0; and 


3. the stationarity condition, Equation (3.27), obtained from Hr =0 


over the time interval [0, tz]. ll 


3.2.3 Generalization 


For the simplest optimal control problem with m control functions and n state 
variables in general, the objective functional to be optimized is 


te 


[= [Pate tm, U1, thy cy Um) dt (3.31) 
ee a 
0 


E ut 
subject to n differential equation constraints 
Yi = gi(y,y,u) or Gi¡=-Y4+9=0; ¿=1,2,...,n (3.32) 
with the n initial conditions 
w(0)— yo, ¿=1,2,...,n (3.33) 


Using the Lagrange Multiplier Rule, the augmented functional is 


te 


m te 
1] P+ SAG; at= [ iss n X) at 
i=l 0 


0 


which was introduced in Section 2.5.2 (p. 44) using the same vector notation. 
At the optimum of J, 6J is zero and is given by 


te ñ m 5 " 
J X fuóvi t M adi + us du + fuð dt=0 
0 i—1 i=1 i=l i=1 
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where 
fy = Fy + $ N05 fu = Y, AjGjy, = 2 Aj, and 
j=l j=l j=1 
fu; = Fu; T AjGi,. 
j=1 


The Necessary Conditions 


Extending the optimal control analysis (Section 3.2.2, p. 59) to the generalized 
problem with multiple states and controls, and defining the Hamiltonian as 


H=F+ 5 Ii (3.34) 
i=1 
we get the necessary conditions for the optimum of J, which are provided in 
Table 3.2. 


Table 3.2 Necessary conditions for the optimum of / in the general problem defined 
by Equations (3.31)-(3.33) 


1. The state equations 
with the initial conditions 
for î = 1,2,...,n 


. The costate equations di = 


with the final conditions A;(tf) = 
fori=1,2,...,n 


. The stationarity condition Au, 


i 


fori—1,2,...,m 


Example 3.4 
Find the necessary conditions for the maximum of 
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subject to the satisfaction of the state equations 


& = —kax%y? (3.40) 
Ne 
gı 
y = — kbr" y? (3.41) 
— n 
92 
where T' is the control and 
E 
k=k —— 3.42 
0exP == (3.42) 


The initial conditions are 
z(0) = xo and y(0) = yo 


This is the batch reactor control problem (seep. 2) without the inequality con- 
straints. Referring to Table 3.2, x and y are, respectively, the state variables 
yı and yg. Furthermore, T is the control u. Using Equation (3.34) and the 
functions F, g1, and ga as indicated above, the Hamiltonian for this problem 
is given by 
H = kerry?  A((—kaz^yP) + A(-kbx*y) 
N — — —À — 
F gi g2 


From Table 3.2, the necessary conditions for the maximum are 


1. the state equations, i = Ha, and y = H»,, given respectively by Equa- 
tions (3.40) and (3.41), and their initial conditions; 


2. the costate equations given by Ài = —H and dd = —H,, ie. 


A = —kaa ty’ (c — aM — b2) 


dg = —ka*^ by*-1(c — aM — bAg) 
and their final conditions A+ (tf) = A2(tf) = 0; and 


3. the stationarity condition H7 = 0, i.e., 


kE 
AS =aM — bà2) = 0 


throughout the time interval [0, te]. ll 
Example 3.5 


The change in reactant and catalyst concentrations z and y, and temperature 
T in a Constant (volume) Stirred Tank Reactor or CSTR without time delays 
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is given by (Ray and Soliman, 1970) 


c= lar — x) — key (3.43) 
aa — (3.44) 
"E AH hA 4- K(T — To) 


where V, AH, p, and C, are, respectively, the reactor volume, heat of reaction, 
liquid density, and specific heat capacity. The subscript f denotes the feed 
property, k is given by Equation (3.42), and Q, m, and K are the three 
controls. They are, respectively, the volumetric flow rate, mass flow rate of 
the catalyst, and the gain of temperature controller as functions of time t. 
The initial conditions are 


z(0) = zo, gy(0)- yo, and T(0)= To 


It is desired to minimize the deviation of the state of the CSTR with minimum 
control action, i.e., to minimize 


where the subscript s denotes the steady state value. Find the necessary 
conditions for the minimum of /. 
Using Equation (3.34), the Hamiltonian is given by 


H = (z— zc) + (y— ys)? -(T — T? + (Q— Qs)? + (m — m 


+A1 © (er — 2) kay + A mw 
Q AH hA+K(T-T,) 
is Š- T) ul LETE i TT, 
TaT ae pap VpC; ( ) 


In this example, x, y, T, Q, m, and K correspond, respectively, to y1, Ya, Y3, 
u1, U2, and uz of Table 3.2. From the table, the necessary conditions for the 
minimum are 
1. the state equations, Equations (3.43)-(3.45), respectively given by 
¿=Hx, y=Hx, and T= Hy, 


with their initial conditions z(0) = zo, y(0) = yo, and T(0) = To. 
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2. the costate equations 


À Q AH 
Ài--H,--3(z-3,-X Š +ky +3 ——k 
1 (x — zs) + Ai y tky dro y 
j Q AH 
Y kE 
js = -Hr = XT — To) + yey 
QAO AH RE, hA KQT— T, T.) 
"Y pCp Ru V pCp 


with the final conditions A;(t¢) = 0, i = 1, 2,3. 


3. the stationarity condition for each of Q, m, and K, i.e., 


Ho = 2(Q +7 LE + a E =0 
Hy, =2(m—m) + 2 =0 
throughout the time interval [0, te]. [] 


3.3 Solving an Optimal Control Problem 


Although a little bit early, it is worthwhile to know how we can actually 
solve an optimal control problem. As indicated earlier, the answer lies in the 
necessary conditions we have established above. They must be satisfied by 
the optimal control functions. 

It must be noted that the necessary conditions are frequently nonlinear 
and cannot be solved analytically. Therefore, optimal control problems are 
generally solved using numerical algorithms, the focus of Chapter 7. 

At this moment, we outline a simple numerical algorithm, which begins by 
guessing the control functions, and involves the following steps: 


1. forward integration of state equations using the initial conditions 


2. backward integration of costate equations using the final conditions 
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3. improvement of each control function utilizing the gradient information 
from H,, — the variational derivative of the objective functional with 
respect to the control function u;. 


The improvement in control functions causes the objective functional value to 
get closer to the optimum. "Therefore, iterative application of the above steps 
leads to the optimum, i. e., the optimal functional value, and the corresponding 
optimal control functions. While the state and costate equations are satisfied 
in each iteration, the variational derivatives are reduced successively. They 
vanish when the optimum is attained. 

An application of the above algorithm to the batch reactor problem of 
Example 3.4 results in the optimal control temperature (T), states (x and 
y), and costates (Aj and A3) versus time, as shown in Figure 3.1 for the 
parameters listed in Table 3.3. The corresponding maximum value of the 
objective functional I is 6.67 kmol/m?. 


H^ = al 
Rote A vendes 33 
e (I = 6.67) - 
750 N 
E 
E 600 
450 


0 10 20 
t (min) 


Figure 3.1 The optimal control, states, and costates versus time in Example 3.4 


Table 3.3 Parameters for the problem in Example 3.4 


a 1.5 te 20 min 
b 1 E 7x10” J/kmol 
c 1 ko 6 x 10% (kmol/m?)- 15 /min 
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3.3.1 Presence of Several Local Optima 


When solving an optimal control problem, it has to be kept in mind that 
several local optima may exist. Consider for example a problem with a single 
control function. The objective functional value may be locally optimal, i. e., 
optimal only in a vicinity of the obtained optimal control function. In another 
location within the space of all admissible* control functions, the objective 
functional may again be locally optimal corresponding to some other optimal 
control function. This new optimal objective functional value may be better 
or worse than, or, even the same as the previous one. 

For example, in the problem of Example 3.4 solved above, the optimum 
obtained is local. Three different optima are obtained when the same problem 
is solved by initializing the algorithm with different guesses for the control 
function. Figure 3.2 shows the corresponding optimal control functions Ti, 
T5, and T3. They are quite different from not only each other but also the 
optimal T in Figure 3.1. While T3 and T> yield the same maximum as obtained 
with T, T3 provides a better maximum of J = 6.70 kmol/m?. 

The above outcome is not unexpected since it is based on the necessary 
conditions for the optimum. As shown in Figure 3.3, the necessary conditions 
are the consequence of an optimum, whether local or global. The satisfaction 
of these conditions does not guarantee the global optimum but merely pro- 
vides a candidate for it. It is the satisfaction of the sufficient conditions that 
guarantees the global optimum. 


T (K) 


Figure 3.2 Three other optimal control functions for the batch reactor problem 


* Satisfying the state equations and any other specified constraints. 
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I an optimum: necessary 


minimum conditions 
(I>!) 
or 
maximum 


PES 


sufficient Í a global optimum: 


conditions 
minimum 
Or 
maximum 


Figure 3.3 Necessary and sufficient conditions for the optimum of the objective 
functional I 


Naturally, we would like to determine the optimum that is global, i. e., valid 
over the entire space of admissible control functions. This quest urges us to 
inquire into the conditions that are sufficient for the global optimum, i. e., the 
optimum over the entire space of admissible control functions. 


3.4 Sufficient Conditions 


For the minimum solution to be global, a sufficient condition must ensure 
that the objective functional value is the lowest possible. In the context of an 
optimal control problem, a sufficient condition for the minimum should lead 
to the inequality 


I(y,u) > I($, à) 


where I is the objective functional, y is the state vector corresponding to any 
admissible control vector u, and y is the optimal state vector corresponding to 
the optimal control vector à. The states, controls, and costates are functions 
of time t, which is the independent variable. The above inequality is opposite 
if the maximum of / is to be determined. 

To yield the above inequality, a sufficient condition must include at least 
one inequality, which in fact is another optimal control problem. Thus, for the 
global minimum in the simplest optimal control problem, a set of sufficient 
conditions by Mangasarian (1966) requires that 
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1. the controls satisfy the necessary conditions for the minimum, 


2. the costates at the minimum are non-positive if g [i. e., the vector of gis 
in Equation (3.32) on p.67] is nonlinear with respect to the states or 
controls, and 


3. the functions F [Equation (3.31), p. 67] and g are convex jointly with 
respect to the states and corresponding controls 


The proof of this result is provided in Appendix 3.C (p. 81). 


The third requirement of Mangasarian gives rise to inequalities of the form 
L(y,u) > 0 [see Inequality (3.52), p. 82]. Proving it becomes a new optimal 
control problem in which zero needs to be established as the minimum of L 
for all admissible u. This task cannot be accomplished analytically except 
in rare, simple cases where the Hessian of L could be easily checked for pos- 
itive definiteness. Because of this situation, the application of any sufficient 
condition in optimal control is very limited. 


3.4.1 Weak Sufficient Condition 


We will now examine the role of the second variation in identifying the opti- 
mum. Let us say we desire to find the minimum of J, and we have determined 
the minimum from the necessary conditions. Then from Equation (2.33) on 
p. 52, and noting that ôI = 0 at the minimum, we get 


I(y,u) — 19,8) = 56 T9; y) + aly) (3.46) 


where ó?I is the second variation of I, and e is the error term, both depending 
on du = (u — à) and the corresponding dy = (y — y). Observe that the non- 
negativity of 21 does not assure the minimum. For example, there may exist 
a u for which 627 is less than the absolute value of e2, which happens to be 
negative. In such a case, let ea = —f where P > 0. Then the right-hand side 
of Equation (3.46) becomes 


ôI 
3 Te < lo Ep < 0 


The above result implies that 7(y,u) — I(y, à) < 0 so that I(y, à) is not a 
minimum! We need to impose some condition on 627 to ensure its minimum. 
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The Condition 


If we place a condition that the error ratio €2/|]5u|]” vanishes with ||óu||, and 
321 > al|Su||? where a is some positive number, then 


V 


"e Q 
I(y,u) I(y,à) ei 3 Ióu|P + €2 


sull? | La. 2 
oul] (i 


If we can find a sufficiently small neighborhood of ú such that all du therein 
have norms sufficiently small to render any negative error ratio less than a/2, 
then 


IV 


I(y,u) —I(y, a) > 0 


in that locality, and I(y, à) is the local minimum. 

The above sufficient condition is weak, since it is applicable only in a suffi- 
ciently small vicinity of the optimal control. The satisfaction of this condition 
does not guarantee a global minimum. Moreover, it is not easy to come up 
with a suitable a in an optimal control problem. 

From a practical viewpoint, the second variation can be utilized 


1. as another necessary condition for the optimum 


2. to speed up the convergence of an optimal control algorithm near the op- 
timum when the variational derivatives of J with respect to u approach 
zero. 


Hence, to increase the confidence on the final optimal solution, the opti- 
mal control problem needs to be solved with different initial guesses to the 
numerical algorithm. If several optima are obtained, then the most optimal 
among them is adopted. There is no golden rule to ensure that the optimum 
is global. 


3.5 Piecewise Continuous Controls 


The optimal control analysis developed above is also applicable when a control 
function is piecewise continuous with a finite number of jump discontinuities, 
as shown in Figure 3.4. Observe that a jump discontinuity suddenly changes 
y or the slope of the state variable. The result is a corner in the graph of y(t) 
at the time of discontinuity where y is not defined. Geometrically, there is no 
unique slope of the curve at a corner. One can draw an infinite number of 
tangents there. 
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jump 
discontinuity 


Uti ta 


jump : 
discontinuity | a 


ti to 


corner 


0 t te 
Figure 3.4 A piecewise continuous control uz,,:. and the accompanying state y(t) 


Let us use uz, st, to denote a piecewise continuous control with jump dis- 
continuities at times tı and t2. Thus, uz, is made up of three continuous 
controls — u1, u2, and uz, as shown in the figure. Then the problem of finding 
the Utta that minimizes 


ty ta 


te 
T= | Fou, i) t Or J Fossa f rua ue În (y, uz) dt 
0 


ty 
Se 0 —— RÀ —"Á NN — 


Ii I2 Is 
subject to 
Y = g (Y, Utta) y(0) = yo, Os<tst 
is the same as finding the minimum of I, I2, and I3 over the contiguous 
time intervals [0, £4), [t1,t2), and [t1, tf] during which the control has no jump 
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discontinuities. Therefore, to find the control us, 4, that minimizes J, we apply 
the developed optimal control analysis to the following three subproblems: 


1. Find the ua that minimizes Jı subject to 
y =9(y,u), yO)=y, 0<t<t 
2. Find the uz that minimizes Jj subject to 
y=9(y,u), t1<t<ta 
where y(t,) is obtained from the previous subproblem 
3. Find the uz that minimizes /3 subject to 
ý =gly,u), t2<t<te 
where y(t2) is obtained from the previous subproblem 


At the minimum of J, additional conditions called, Weierstrass-Erdmann 
corner conditions, must be satisfied on the corners. These conditions require 
the continuity of the Hamiltonian and costate at the corners. The details will 
come later in Section 6.6. 


3.A  Differentiability of A 


We need to show that A is differentiable with respect to t in the following 
equation on p. 62: 


te tf 
fa, +AGyjóyar— [ due =0 (3.16) 
0 0 
so that 
te te 
j (Fy  AGy)óy dt — [Ayl e, + [Abu], s + / uo (3.17) 
0 0 


Applying integration by parts to the first integral in Equation (3.16), we 
obtain 


t te $ t y. 
ay | (F, + AG,) dt -f öf (E, 4- AG,) dt at— f rópat=0 (3.47) 
0 0 0 
0 0 
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Introducing 
t 


o(t) = J (Fy +AGy) dt 
0 
and considering that dy is zero at t = 0 due to the initial condition [Equa- 
tion (3.6), p. 58], we can write Equation (3.47) as 


te 


by (ts) (te) — / (é(t) + jog dt =0 (3.48) 
0 


Note that the above equation is valid for the entire class of continuous dy 
functions that are zero at t = 0. This class includes the subclass of dy func- 
tions, which are zero at t = tr as well. Now if dy belongs to the subclass, then 
the coefficient of dy in Equation (3.48) is a constant, as we will show shortly. 
This is a very important result, which implies that the coefficient [¢(t) + A] 
has to be the same constant for the entire class of óy functions. If that is not 
so, then Equation (3.48) will obviously not apply to the subclass and therefore 
will not be valid for the entire class of óy functions. 

Before examining the differentiability of A, let us show the constancy of 
the coefficient of dy, i.e., [p(t) + A]. This result is known as the Lemma of 
Dubois-Reymond. 


Constancy of the coefficient of ój 
With dy additionally being zero at t = tf, Equation (3.48) becomes 


tr te 


/ (a(t) + AJógdt = | did sso (3.49) 
a mua 

0 c(t) 0 
where c(t) stands for the coefficient of dy. Let cy be the average of c(t) in the 


interval [0, ts]. Then 


te te 


Jena BR a n 


0 0 


which yields upon rearrangement 


'Thus, the choice of 


80 Optimal Control for Chemical Engineers 
conforms to the current conditions of dy being zero at t = 0 and tf. For this 
choice, dy is given by 

dy = c(t) — Co (3.50) 


and is admissible. We will use it in the following result 


te te 


fico aia | osia - a Bj ; =0 (3.51) 
0 0 


which is due to Equation (3.49) and the conditions dy(0) = dy(ts) = 0. Sub- 
stituting the dy from Equation (3.50) in Equation (3.51), we obtain 


te 


I [e(t) — co]? dt = 0 


0 


Observe that the above integrand, being a squared term, cannot be negative. 
It must be zero throughout the interval in order to satisfy the above equation. 
Thus, 


c(t) 2c, 0<t<tr 
showing that c(t) or [p(t) + A], the coefficient of dy in Equation (3.49), is a 
constant. As explained earlier in the paragraph following Equation (3.48), 
this result extends to the entire class of dy functions. Thus, [ó(t) +A] is a 


constant in Equation (3.48) for all óy functions. Let us now examine the 
differentiability of A by expressing the above result, i. e., 


i 
Olt) A fy AG) +A = a 


Or 


-- fer, + AG) dt + co 
0 


The above equation shows that A is differentiable with respect to t at the op- 
timum for which Equation (3.16) is the necessary condition. At the optimum, 
the derivative of A with respect to t is 


À = —(F, + AG,) 
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3.B Vanishing of (F, - AG, +14) at t=0 


We need to show that the coefficient (Fy + AG, + A) of dy in the following 
equation on p. 63: 
te 
/ F; 4- AG, +A óydt — 0 (3.20) 
0 


has to be zero at t = 0 even though dy there is zero. 

Suppose that the coefficient (F, + AG, + À) is greater than zero at t = 0. 
This supposition means that (Fy + AG, + À), which is a continuous function,“ 
has to be greater than zero in a finite subinterval [0, tı] however small. Now 
dy could be any continuous function provided it is zero at t = 0. Satisfying 
this provision, we choose a dy to be t(tı —t) in the subinterval but zero outside 
it. Figure 3.5 shows these two functions as well as their product, the integral 
of which is 


te ti 
| 59 dy dt = | F, AGy +À t(& — tdt > 0 
0 0 


in contradiction to Equation (3.20). The above integral turns out to be posi- 
tive because (Fy + AG, +A) is positive in the subinterval [0, tı], and t(t — t) 
is zero at t = 0 but positive otherwise. 

Supposing the coefficient (F, + AG, + A) at t = 0 to be less than zero 
contradicts Equation (3.20) again. The integral turns out to be negative this 
time. 

Thus, neither greater nor less than zero as supposed, the coefficient of dy, 
i.e., (Fy + AG, + A), has to be zero at t = 0 where dy is already zero. This 
result is due to Du Bois-Reymond (1879) and holds for any t where dy is zero. 


3.C Mangasarian Sufficiency Condition 


For the minimum (maximum) of the optimal control problem given by Equa- 
tions (3.31)-(3.33) on p. 67, it is sufficient to have 


1. the control à satisfy the necessary conditions in Table 3.2, 


* Because of the assumption made at the outset that the partial derivatives of F and g with 
respect to y are continuous. 
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(Fy + AG, + A)óy 


te 


Figure 3.5 The functions (F, + AG, + À), óy, and their product, i.e., the inte- 
grand of Equation (3.20) 


2. all non-positive (non-negative) elements of the costate vector A in the 
interval [0, tr] at the minimum (maximum) if 


7 
g= gı 92 ee Jn 


is nonlinear with respect to u or y, and 


3. a convex (concave) F as well as g jointly with respect to u and the 
corresponding y in the interval (0, te] 


Note that the entities H, F, g, and their partial derivatives are functions of 
y and u. Such an entity, say, H, is denoted by H when evaluated at the 
optimum, i.e., for the vector of optimal controls à and the corresponding 
vector of optimal states y. 

Let us show the result for the minimum of I or Î.* Because F and g are 
specified to be convex and continuously differentiable, 


F-F>E (y-9)+£ (u—ú) (3.52) 
g-&>8, (y-y) +8, (u- û) (3.53) 


throughout the time interval [0, tr]. 


* To show it for the maximum, reverse the inequalities that follow. 
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Since ú satisfies the necessary optimality conditions (see Table 3.2, p. 68), 


A=-Hy => Fy=-A\-BA => Fy =-AT-AlBy (3.54) 


A, =0 > Fa=-£lA > F 


u 


= —A Su (3.55) 
Using Equations (3.54) and (3.55) in Inequality (3.52), and integrating both 
sides provides 
te te 
I-Î > — / À! (y —y)dt- J A &y(y —¥) +A ulu — à) dt (3.56) 
0 0 
Integrating by parts the first integral in the above inequality, we get 
te te te 
[i-a [i-ar [Te-a (857 
0 0 0 


where we have applied the final costate condition A(tf) = 0, the non-variation 
of the state at t = 0, i.e., y(0) — y(0) = dy(0) = 0, and state equations in the 
last step. With the help of Equation (3.57), Inequality (3.56) becomes 


te 
I-12 fats & - &y(y 9) —u(u— à) dt 
0 


in which the square-bracketed term is greater than zero due to Inequal- 
ity (3.53). Because the costate at the minimum is specified to be non-positive, 
i.e., A > 0, we obtain J — Î >0. 

If & is linear with respect to u and y, then Inequality (3.53) becomes an 
equation so that the specification A > 0 is not required to obtain J — Î>o0. 
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Exercises 


3.1 Find the necessary conditions for the minimum of 


subject to 


jı = -yu + yw’, y1(0) = y 
Yo = you, y2(0) = 0 


3.2 For the plug flow reactor problem on p. 6, find the necessary conditions 
for the minimum of 


7 d 
I= J SY ay (1.12) 
FA 
0 
subject to 

dy —kiyP | Ako(yo — y)? P? 
Z =+8 plc 3.58 
dz 2yo — y (2yo — y)? SES 


with y(0) — yo. 
3.3 Find the necessary conditions to maximize the profits 


I = ys (tr) 


from an isothermal CSTR carrying out four simultaneous reactions described 
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by (Jensen, 1964) 


Yi = u4 — (ui + U2 + u4)yi — 01Y1Y2 — a2yiyeus 
Yo = ui — (u1 + ua + u4)ya — aryl y2 — 203Y2Y3 
Ya = U2 — (u1 + ua + u4)ya — aayays 


Ya = — (u1 + U2 + u4)ya + 2a1yiya — 04YaYs 
Ys = —(u1 + U2 + u4)ys + 05Y2Yz — 04YaYs 

Yo = — (ua + uz + u4)ye + 2aAyays — 02Y1Y6U3 
Y7 = —(ui + u2 + u4)yz + 2a2yiyeua 


Ys = as[(u1 + ua + U4)Y1 — Ua] — agui — azua 
+ (us + ua + u4)(a2Ya + agys + agys + a10y7) — anu — 012 

with initial conditions y;(0) = yio for i = 1,2,...,8 over the time interval 
[0, £c]. The coefficients a, to a2 are constants. The controls u; to ua are, 
respectively, three feed flow rates and electrical energy input to the CSTR as 
functions of time. 
3.4 Show that the non-autonomous problem of finding the minimum of 

te 

1= | Fio ut. da 
0 


subject to 
Gly(t), y, u(t),t) = —9 + gly(t), u(t), t] = 0 
is similar to the autonomous problem in which £ does not appear explicitly as 
an argument of F and g. 
Hint: Propose t as a new state variable, say, yo (see Section 9.7.1, p. 270). 
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Chapter 4 


Lagrange Multipliers 


In this chapter, we introduce the concept of Lagrange multipliers. We show 
how the Lagrange Multiplier Rule and the John Multiplier Theorem help us 
handle the equality and inequality constraints in optimal control problems. 


4.1 Motivation 
Consider the simplest optimal control problem, in which we wish to find the 
control function u that optimizes 
te 
[= [row dt (3.4) 
0 


subject to the differential equation constraint 


Y = 9(y,u) or G(y,y,u) =—yt+gly,u) =0 (3.5) 


with the initial condition 
y(0) = yo (3.6) 
At the optimum, it is necessary that the variation given by 


te 
ôI = eu + Fusu) dt 


0 


is zero, while satisfying the differential equation constraint. Because the con- 
straint ties y and u together, we cannot have dy or du arbitrary and thus 
independent of each other in the above equation. Recall from the last chapter 
that when the variations are arbitrary, their coefficients are individually zero, 
thereby leading to the necessary conditions for the optimum. This simplifica- 
tion is, however, not possible with the above equation. 

Dealing with this problem is easy if Equation (3.5) could be integrated to 
provide an explicit solution for y in terms of u. Then one could substitute 
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y = y(u) into Equation (3.4) and obtain J in terms of u alone. However, this 
approach fails in most problems where analytical solutions of the involved 
constraints are simply not possible. 


4.2 Role of Lagrange Multipliers 


The above difficulty is surmounted by introducing an undetermined function, 
A(t), called Lagrange multiplier, in the augmented objective functional defined 
as 


1=| F(y,u) + AG(y,y,u) dt (3.7) 
0 


At the optimum, the variation of J is 
te 
óJ = "nc + F,du + AG,0y + Gy0y + Gu6u) + Gor] dt = 0 (4.1) 
0 


where the role of A is to untie y from u by assuming certain values in the inter- 
val [0, tr]. Given such a A, we are then able to vary dy and du arbitrarily and 
independently of each other. This ability leads to the simplified necessary con- 
ditions for the optimum of J and, equivalently, for the constrained optimum 
of I. The conditions include an additional equation for A, the satisfaction of 
which enables the arbitrary variations in the first place. 

In Section 3.2.1 (p. 59), we had asserted the Lagrange Multiplier Rule that 
the optimum of the augmented J is equivalent to the constrained optimum 
of I. This rule is based on the Lagrange Multiplier Theorem, which provides 
the necessary conditions for the constrained optimum. We will first prove this 
theorem and then apply it to optimal control problems subject to different 
types of constraints. 


4.3 Lagrange Multiplier Theorem 
The theorem states that if a functional I(y) subject to the constraint 
K (y) = ko 


is optimal at y = j near which ôI and ôK are weakly continuous, and there 
exists a variation 6z at y along which 


óK(;óz) =0 
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then the following conditions are necessary: 


1. 


2. 


The constraint is satisfied, i. e., 


K(9) = ko (4.2) 


There exists a constant multiplier A such that 
SI (9; Oy) + ASK (9; dy) = 0 (4.3) 


for all variations dy. 


Remarks 


1. 


The provision that ôK (3; 62) = 0 along at least one variation dz is called 
the normality condition or the constraint qualification. 


. Equations (4.2) and (4.3) are the necessary conditions for the con- 


strained optimum of I at j. 


. Both / and K in a general case are functionals of a function y depending 


on an independent variable. 


. Weak continuity of ôI means the continuity of 91 with respect to y for 


each fixed dy. This is a relaxed requirement. Not fixing dy would have 
imposed the demand for the Fréchet differential instead of the variation 
of I. 


. If ôI is weakly continuous near 9, it means at any y near j, ôI is contin- 


uous for each fixed dy. In other words, if y; and y» are two functions in 
the vicinity of y, then ôI (y1; dy) approaches 6I(y2; dy) as yı approaches 
ya for each fixed dy. 


Example 4.1 


Consider the functional 


whose variation is given by 


tr 


i= | 5o) dy dt 


0 


Then the weak continuity of ó/ means the continuity of the partial 
derivative Fy. 
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Note that most of the optimal control problems have continuous partial 
derivatives of the involved integrands and, therefore, satisfy the require- 
ment of weak continuity. 


Outline of the Proof 


As shown in Figure 4.1, together with the givens and an assumption of a non- 
zero Jacobian of / and K at the optimum, we will invoke the Inverse Function 
Theorem (Appendix 4.A, p.115) to contradict the optimum of J. This step 
will entail the Jacobian being zero and lead to the desired conclusion. 


at y: 
Givens Inverse function zd I is NOT optimum at y 
theorem (contradicts Given 1) 
1. I is optimum 
2. K — ko V 
3. ôI and ôK are The assumption is wrong! 


weakly continuous The Jacobian is zero at y 


| 


Assumption K Shiau 
Non-zero Jacobian of ôI + AK =0at y 
I and K (necessary conditions for 


the constrained optimum) 


Figure 4.1 Outline of the proof for the Lagrange Multiplier Theorem 


The Proof 


Let y be the optimal function. Since any admissible function must satisfy the 
the constraint K (y) = ko, we have 


K(9) = ko (4.2) 


as the first necessary condition for the constrained optimum of I. 
Consider this ĝ and its arbitrary variations, dy and óz. Keeping these three 

functions fixed, the functional values J(j-4- uóy-4-vóz) and K(--uóy-4-vóz) are 

functions of y and v, which are some real numbers. Let these two functions 
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be, respectively, 
i(p, v) = I(ĝ + poy + vóz) (4.4) 
and k(p,v) = K(G + dy + vóz) (4.5) 
for all y and v within a region R that is sufficiently small to allow for the 
weak continuity of ôI as well as ôK at y = (9 + uy + vz). Figure 4.2 shows 


the (u,v) coordinates in R mapped to the (i,k) coordinates by ¿(1, v) and 
k(u, v). 


Jacobian Determinant 
We now introduce the Jacobian determinant of i and k 


lu ty 


D E 
(us v) ky, kp 


where î,, for example, is the partial derivative of ? with respect to y and is 
given by 


i(u + o, v) — (1, v) 


es: a 
srir a td E S A | 
a—0 a 


d 
= —1(9 + uóy + vôz)a=0 
da 


= ôI (ĝ + uóy + văz; dy) (4.6) 


Figure 4.2 Mappings between (ji, v) and (i, k). The shaded region has the inverse 
mappings. 
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The last equation follows from the definition of the variation [see Equa- 
tion (2.12), p.36]. Similar results are obtained for the remaining partial 
derivatives. Using them in the definition of D, we obtain 


O1(G + poy + vdz; dy) SIl + poy + voz; 0z) 


D = 
DATI a jale) 


Note that the given weak continuity of ôI and ôK gets translated into the 
continuity of their equivalents, i. e., the partial derivatives of i and k. In other 
words, the functions 7 and k are differentiable in the sufficiently small region 
containing u and v. 

When y = v = 0, the Jacobian becomes 


D(0,0) = SI dy) 010308) 
dK(9;0y) 6K(9;02) 

which is at the optimum. Let us assume that D(0,0) is not zero. This 
assumption rules out dy or 62 being zero throughout the t-interval [0, tr]. 
Otherwise, say, if óy is zero throughout the interval, then both terms of the 
first column of the Jacobian would be zero and result in D = 0. 


Application of Inverse Function Theorem 


Observe from the definitions in Equations (4.4) and (4.5) that y = 0 and 
v = 0 correspond to i(0,0) = I(j) = 2, and k(0,0) = K(9) = ko using 
Equation (4.2). Figure 4.2 shows these coordinates. The functions i(u, v) and 
k(p, v) are differentiable in the region R around y = v = 0. 

Since D(0,0) is not zero, the Inverse Function Theorem (Appendix 4.A, 
p. 115) guarantees the existence of inverse functions 


p=? (5, k) (4.7) 
y = k (i, k) (4.8) 


which are continuous in some region shown shaded around (i, ko) in the figure. 
In this region, we can always pick a coordinate (i1, ko) where ¿1 is less than 
I(g) and the constraint K = ko is satisfied. Then Equations (4.7) and (4.8) 
provide 

Hi = iY (ii, ko) and WI = k (i1, ko) 


which contradict the specification that i is the minimum. The presence of 
uj and vı implies that while satisfying the constraint K(y) = ko, a control 
function y = (ij + u40y + 1102) provides i; = I(y), which is less than î = I (i). 
Since this is impossible, our assumption cannot be true, so that the D(0,0) 
has to be zero. Therefore, 

ól(j;óy) 01(9;02) 


Rte OK (Gj; dy) K (y; 6z) zh a 
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which leads to the second necessary condition for the minimum of 1 (y) at y 
subject to the constraint K (y) = ko. The same condition is obtained for the 
constrained maximum of I by considering i4 greater than I(ĝ). 

Expanding Equation (4.9), we get 


ÒI (9; 0y)0K (3; 62) — 01 (9; 02)0K (9; dy) = 0 (4.10) 
for all functions óy and óz. With the introduction of 


01 (9; 02) 


à= -SK (9,02) 


where ôK (ĝ; 62) is not zero as per the given constraint qualification, Equa- 
tion (4.10) becomes 


Ilg; dy) + AK (9; dy) = 0 (4.3) 


for all óy and a Lagrange multiplier A. The above equation is the second 
necessary condition for the constrained optimum of I(y) at y subject to 
the constraint K (y) = ko. This completes the proof of the Lagrange Multiplier 
Theorem. 


Lagrange Multiplier Rule 


This rule is based on the Lagrange Multiplier Theorem. Consider the aug- 
mented functional 
J(y) = Ty) + A[K (y) — ko] 


where both y and A are functions of an independent variable t. According to 
the theorem, the necessary condition for the optimum of J at y is that 


ôJ(ĝ;ðy)=0, or 


SI (3; by) + ASK (9; dy) + SALK (9) — ko] = 0 
Ne — ua Ne — d 
Ji Ja 


Since 6 is arbitrary and appears only in the last term J2 of the above equa- 
tion, both Jı and Jz should be individually zero. Thus, 


Ja = GALE (9) — ko] = 0 


implies that 
K(9) — ko (4.2) 


because 6A is an arbitrary function of t. Next, Jı = 0 yields 
01(9;0y) + AK(9; dy) = 0 (4.3) 


According to the Lagrange Multiplier Theorem, Equations (4.2) and (4.3) 
are the necessary conditions for the optimum of J subject to the following 
preconditions: 
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1. The variations of both J and K are weakly continuous near ĝ. 


2. The constraint qualification that the variation of K at y = y is non-zero 
for at least one variation of y. 


Observe that Equations (4.2) and (4.3), which arise from óJ(j;óy) = 0, are 
also the necessary conditions for the optimum of J. This fact gives rise to the 
following Lagrange Multiplier Rule: 

The optimum of a functional I(y) subject to a constraint K(y) = ko is 
equivalent to the optimum of the augmented functional 


J(y) = I(y) + A[K (y) — ko] 


where A is an undetermined Lagrange multiplier, both ó/ and 6K are weakly 
continuous near the optimal y, and the constraint qualification ôK (y; 62) = 0 
holds along at least one variation dz at the optimal y. 

Before considering the applications of the Lagrange Multiplier Theorem, it 
is worthwhile to generalize the theorem to handle several equality constraints 
and functions. 


4.3.1 Generalization to Several Equality Constraints 


Let us derive in terms of Lagrange multipliers the necessary conditions for the 
optimum of I(y) at y = y subject to two equality constraints 


Ki(y) = ki and K»(y) = ko 


and weak continuity of 91, 0 K4, and dK near ĝ with the following constraint 
qualification: 
There exists at least a pair of variations 62, and 6z2 corresponding to which 
the determinant 
0K1(9;021) 0K1(9;022) 


óK»(i;0z1) 9K2(9;022) 


If y is optimal, then it must satisfy the above constraints. Therefore, the 
first set of two necessary conditions is 


Ki(d) = kı and K»(d) = ko 
Now consider a function 
y = Y + poy + 11021 + voóza 


in the vicinity of y with three arbitrary but fixed variations, dy, 021, and 
zə. The variables u, vı, and va are some real numbers in a region R. Then 
the functional values I(y), Ki(y), and Ko(y) are the functions i(j 14, V2), 
ki (p, v1, v2), and ka(p, 1, v3), respectively. If R is small enough to allow for 
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weak continuity of ôI, 6K1, and óK5 near j, then from Equation (4.6) it 
follows that 


Oi Oi ði 
On n OV, 
Ok, Ok, Ok 
D(u,v1, v2) = En pu. Cm 
Ou OM Ov, 


ól(yióy)  SI(y:8z1) — 6I(y;622) 
=|9K1(y;0y) óKi(y;óz) 9K1(y;0z2) 
dKo(y;0y) 9K2(y;021) óK»(yióz2) 


Applying the Inverse Function Theorem as before, we establish 


5I(9;5y) SI; zı)  ó1(9;022) 
D(0,0,0) =| 641 (9; ðy) 09K1(9;021) 9K1(9;0z2)| =0 

6K2(9;6y) ¿K2(9:021) 5K 2(9; 22) 

which is the remaining necessary condition for the optimum of I(g) subject 

to Kı(ĝ) = kı and K2(9) = k2. This condition expands to 

ól(jiózi)  6I(9;622) 

6K 2(9; 621) 6K2(9; 622) 
—— £ÜÁ 


9K1(9,021) Kı (9; 622) 
6K (9; 621) 0K2(1; 022) 


AA E 
no nı 


91 (9; 021) 91 (4; 622) 
9K1(9;0%1) AK1 (9; 022) 
O ———À 


ôI (4; dy) — ôK: (f: dy) 


+ óK»(j; by) =0 


72 


where 79, 71, and na are the determinants as indicated above. 
Because of the provision of constraint qualification, po = 0. Thus, we can 
introduce the Lagrange multipliers 


A E —m/no and Az = m2/mo 


to obtain 


SI (9; dy) + MOK1(9; dy) + A26K2(9; dy) = 0 


as the final necessary condition for the constrained optimum of I. 
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Lagrange Multiplier Theorem for Several Equality Constraints 


Generalizing the above results for m constraints, K;(y) = ki, i= 1,2,...,m, 
the necessary conditions for the optimum of 1 (y) at y 


Ki(9) = ki, ¿=1,2,...,m 
SIC; dy) + XC Aio KiC; Sy) =0 
i=1 
subject to the following preconditions: 


1. The variations of J and K;, i = 1,2,...,m are weakly continuous near 
the optimum at %. 


2. The constraint qualification that there exists a set of m variations 
(z1, 022, ..., 62m) 


of y such that the determinant 


Kı (9; 621) 0K1(9;0z2) ... 5K 1 (9; zm) 
dK2(9021) 0K2(9;022) ... 5K 2(%; 52m) 

no = . . i : =0 
IKm(Y; 921) Kf; 622) tee Km (9; 9Zm) 


Lagrange Multiplier Rule for Several Equality Constraints 


Similar to the case of a single constraint in the last section, the rule is as 
follows: 
The optimum of a functional 7(y) subject to constraints 


Kily) = ki, i—1,2,...,m 


is equivalent to the optimum of the augmented functional 
J(y) = I(y) + » ri[Ki(y) — ki] 
i=1 


where A;s are undetermined Lagrange multipliers, 07 and all 6K;s are weakly 
continuous near the optimum, and the aforementioned constraint qualification 
no = 0 holds for at least one set of m variations of y at the optimum. 


4.3.2 Generalization to Several Functions 


The result of the above section is easily extensible to the general case in which 
the objective functional and equality constraints depend on several functions. 
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Let J be a functional dependent on the vector of n functions 


" 
y= y ya ee Un 


and subject to m constraints 
Kily) =k, t=1,2,...,m 
If satisfying the above constraints / is optimum at y = y such that 
1. the functionals SI and all 6js are weakly continuous near y, and 
2. there exists a set of m variations 
(921, ÔZ2, ..., OZm) 


where dz; is the i-th variation vector at y given by 


T 
6z; = Ózii Ózj2 ceo» ÓZin 


for which the following constraint qualification is satisfied: 


K(f; 021) Kı (y; 622) dara K(f; Zm) 
ôKə(f;ðz1ı) óK»(y:;óz2) ... K($; Zm) 

mo = l | l | =0 
dKm(y;021) 6Km(Y;6z2) ... óKa(yióza) 


then the necessary conditions for the optimum are 
Ki(y) = ki, 121,2, .,m 


51(¥; 59) + M] NK, (9:ó$) =0 


i=1 


where A;s are the m undetermined Lagrange multipliers. 


4.3.2.1 Simplification of Constraint Qualification 


The constraint qualification m = 0 can be simplified by expanding the terms 
of the determinant. In general 


OK; aK; OK; 
Ki(¥; 62) = 0251 + 2622 ++ ÓZjn 
óKi(y;02;) on a 6252 + + 5, 6% 

= Kz; 
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where HK; is the vector of partial derivatives of K; with respect to y and is 
evaluated at y — y, i.e., 


Kye Ky 622 Ky 62m 
Kj; Ky 622 K3g0Zm 

no = =0 
Kg 521 Km 3022 sae Kg 62m 


If a determinant is not zero, it means that its rows are linearly independent. 
In other words, none of the rows can be expressed as a linear combination 
of other rows. This result in case of the above determinant mo means that 
there is no K; that can be expressed as a linear combination of other Kj 
(j 21,2,...,m, j — i). 

As a consequence, the simplified constraint qualification is that the vectors 


OK; OK; OK; 


Kiş = s 
y Op Oy» OYn y 


are linearly independent. 


Generalized Lagrange Multiplier Rule 


The rule for this general case of several constraints and functions is as follows: 
The optimum of a functional I(y) where 


yr Yi Ya tee Yn 
subject to m constraints 
Kily) = ki, i—1,2,...,m 


is equivalent to the optimum of the augmented functional 
J(y) = I(y) + 5 Ai[Ki(y) — ki] 
i=1 


where A;s are undetermined Lagrange multipliers, 07 and all 0.K;s are weakly 
continuous near the optimal y, and the vectors of partial derivatives Kiş 
(i = 1,2,...m) are linearly independent. 
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4.3.3 Application to Optimal Control Problems 


We will apply the Lagrange Multiplier Rule to obtain the set of necessary 
conditions for the optimum in an optimal control problem constrained by a 
differential equation. In Section 3.2, we asserted the rule and obtained the 
following necessary conditions (see p. 60): 


G=0 (3.11) 
and 
te 
(OF + Aó6G) dt = 0 (3.12) 
0 
for the optimum of 
te 
I= [row dt (3.4) 
0 


subject to the differential equation constraint 


Y = 9(y,u) or G=-y+gly,u) = 0 (3.5) 


over the interval [0, tf] with the initial condition y(0) = yo. 


Series of Equality Constraints 


We will first show that the differential equation poses a series of equality 
constraints along the t-direction. Then we will apply the Lagrange Multiplier 
Rule for the optimum of / subject to those constraints. 

As shown in Figure 4.3, let y; and u; be the values of y and u at a point t; 
in the interval [0, tf]. Then at each successive t greater than 0, the differential 


Ji 
x 
y in " 
Yo Yi : x 
x x ' | 
uo Ui a : ER Un 
e e . . 
to=0 =— i-th subinterval —=: tn = te 
ti ti 


Figure 4.3 Values of y, u, and g in the i-th subinterval 
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equation constrains the corresponding values of y to 


ti ta tn-1 tn=te 
Yo + n g dt, n+ fgat, 2, Yn-2 + [oct and Yn—1+ | ua 
to=0 ti tn—2 tn-1 
== ee WA A — — — —— 
yi y2 Un-1 Un 


where for each i-th subinterval [t;,t;+1], the length At; = (t;+1 — ti) tends to 
zero. The ¿-th such constraint is 


tii 


l gly, u) dt = yi — yi 


which can be written as 


li At; = Vii — Yi 

Mod Gi 2 Yi+1 Yi 

where from the Mean Value Theorem for integrals (Section 9.14.2, p. 276), gi 
is the value of g at some t in the i-th subinterval. Rearranging the above 
equation, 

_ Vii y 


i 
¡ At; = 0 
At;—0 At; tgi í - 
ki 


A 


Ki(zi) 
where we have denoted the left-hand side as the functional K;. In the limit 
At; tending to zero, K; depends on the vector 


: T 
Zi — Yi Yi Ui 


Hence, for all t > 0 in the interval (0, tf], the differential equation constraint 
is equivalent to the following series of equality constraints: 


Ki(zi) — 0, 1= 1,2,...,n (4.11) 
where n tends to infinity. Note that there is only one constraint K;(z;) — 0 
at a given t = t; in the interval (0, tg]. 
4.3.3.1 Serial Application of the Lagrange Multiplier Rule 


The optimum of J subject to the first constraint Kı = 0 is equivalent to the 
optimum of 
Ji =I+A¡K; 


In turn, the optimum of Jı subject to the second constraint Kə = 0 is equiv- 
alent to the optimum of 


Jo = Jı + A2K2=1+A1K1 + A2K 
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Thus, the optimum of Jz is equivalent to the optimum of J subject to the first 
two constraints. In this way, we sequentially arrive at 


n 
J=J = 14) MK; limn > oo 
i=1 
whose optimum is equivalent to the optimum of I subject to all constraints 
Ki;(zi =0, ¿=1,2,...,n, limn —> oo (4.11) 
Consequently, the necessary conditions for the optimum at Z are 


1. the constraints given by Equation (4.11) or G = 0 along with y(0) = yo, 
and 


2. the equation 


noo 
6I+ M AK; =0 (4.12) 
i=0 
where A; is the Lagrange multiplier corresponding to the i-th constraint, 
Ki(y) = 0, at t = ti. 
We now need to simplify the last condition. The variations ôI and ôK; are 
respectively given by 


ar = [5r and 


A ÓYir1 — OY; 
K; = oe i hi i A 
? At, 30 At; un du + 9 p du; Ati 
ôg 
Inserting them in Equation (4.12), we get 
n—>00 
_ Óyixi — Óyi 
foras Dax lim X oe 
gu 4, 5Yi + Gul ¿Jus At; = 0 (4.13) 


With At; tending to zero as n increases to infinity in the above equation, 


ÓYi+1 == OY; = d 
At; dt 


and the summation term becomes an integral. Incorporating these two results, 
Equation (4.13) becomes 


te 
J ÔF + A(—6y + gydy + gudu) dt = 0 
— s — 


0 óG 
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where A is an undetermined function of the independent variable t. The 
bracketed term multiplied by A is 9G obtained from the definition of Œ. Hence, 
the above equation can be expressed as 


te 
for + A6G) dt = 0 (3.12) 
0 


To summarize, the above equation and 
G=0 (3.11) 


along with y(0) = yo are necessary for the constrained optimum of T. 


Preconditions for the Optimum 


It can be easily verified that the above result is subject to the following pre- 
conditions at each t in (0, ts]: 


1. The variations of / and G are weakly continuous near the optimum. 
Given that Fy = 0 and Gy = —1 are constant and therefore continuous, 
the weak continuity condition means that the partial derivatives of F 
and g are continuous with respect to y and u in the vicinity of the 
optimal pair (9), à). 


2. The following constraint qualification exists. There exists at least one 
set of variations óz = (dy, dy, du) for which the variation of G at each t 
in (0, tr] is not zero at the optimum. 


Example 4.2 
Consider the problem in Example 3.1 (p.65). According to the Lagrange 
Multiplier Rule, the minimum of 


te 
T= fue 
0 
subject to 
y=—-JVytu (3.24) 
with the initial condition y(0) = 0 is equivalent to the minimum of the aug- 
mented functional 
tf 
=f (0 eda tege ht (4.14) 
— _, — —/ 
0 F g 
subject to the initial condition. The Lagrange multiplier À is an undetermined 


function of the independent variable t. This result is subject to the following 
preconditions: 
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1. The partial derivatives of F and g with respect to y and u are continuous 
in the vicinity of the minimum. 


2. The following constraint qualification is satisfied for G = —y+ g. There 
exists at least one set of variations (ý, ôy, ôu) at each t in (0,tr] for 
which the variation of G is not zero at the minimum. 


Example 4.3 
Similarly in Example 3.2 (p.65), the maximum of 
te 
E 
I= fratas, k = ko exp =+5 


RT 
0 


subject to 
t = —akz^, x(0)=x0 


with T as the control function is equivalent to the maximum of the augmented 


functional 
te 


J =| ckz + (—4—-aka*) dt, x(0)=x0 
NR ” “= — 
0 F g 
where A is an undetermined function of the independent variable t. The vector 


ES T ; ; s EE 
Z— $ T provides the maximum under the following preconditions: 


1. The partial derivatives of F and g with respect to z and T' are continuous 
in the vicinity of Z. 


2. The constraint qualification is satisfied for G = —y + g. Thus, there 
exists at least one set of variations (9y,0y,09u) at each t in (0,t;] for 
which the variation of G is not zero at the maximum. 


4.3.3.2 Generalization to Several States and Controls 


Consider the optimization of the functional 
te 


r= [Puya 


0 


where y and u are, respectively, the state and control vectors 


y= y(t) wot) ... ynlt) and 


u= u(t) uat)  ...  Umlt) 
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subject to the autonomous ordinary differential equations 


y = g(y,u) 
with the initial conditions 
y (0) = yo 
where g is the function vector 
ayu) golysu) o myu) 


The above problem is equivalent to optimizing the augmented functional 
te 
=| F+A (-y+8) dt 
— ee” 
0 G 


subject to the initial conditions where A is the vector of time dependent 
Lagrange multipliers 


A(t) A2 (t) es. At) 
'The preconditions for the optimum are as follows: 


1. The partial derivatives of F and g with respect to y and u are continuous 
in the vicinity of the optimal pair (y, à). 


2. The following constraint qualification is satisfied at each t in (0, tg): 


ôG (2; ôz1) dG, (2; 6z2) eed Gu (2; Zm) 
Ga (2; óz1) Ga (2; 023) Vidas Ga (2; OZm) 

no = = 0 
SGm(â; 5621) 9Gm(2;0Z2) ... IGm(2; Zm) 


where z is the coordinate set (y, y, u), Z is the optimal z, and óz; the 
i-th variation (6y;, 6y;, 6u;) in z at 2. 


Example 4.4 
Consider the isothermal operation of the CSTR in Example 3.5 (p.69). The 
reactant and catalyst concentrations, r; and x2, are governed by 


y u 

Ly = yt = 21) = kat, x1(0) = 11,0 
A U2 — u1r2 

Ta = EE cu z2(0) = 22,0 
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where the controls ui and us are the volumetric flow rate of the CSTR and 
the catalyst mass flow rate, respectively. The aim is to minimize the deviation 
of the state of the CSTR with minimum control action, i.e., to minimize 


where the superscript s denotes the steady state value. 
The above problem is equivalent to minimizing the augmented objective 
functional 


te 2 
J= / 5 (zi — 13)? + (uu) +A -tı + (er = 21) — ku 
o0 j=l SS A A NEI 
F gı 
tA -i t Tu a (4.15) 
92 


where 11(0) = 210, £2 = 22,9, and A and àz are time dependent Lagrange 
multipliers. The preconditions are as follows: 


1. The partial derivatives of F, gi, and ga are continuous with respect to 
11, 22, ui, and uz in the vicinity of the minimum. 


2. With G; defined as (—4; + gi) for i = 1 and 2, the following constraint 
qualification holds at each t in the interval (0, tr]: 


ôG (2; ôz1) ôG (2; 6z2) 0 
T = = 
"= ia Ss ardas) 
where z is the coordinate set 
(1, $2, 71, T2, U1, u2) 


2 is the optimal z and dz; the i-th variation set 


(tii, Stoi, 011, Óx2, Guri, duos) 


4.3.3.3 Presence of Algebraic Constraints 


Consider the “base” problem to optimize 


te 
r= | riy at 
0 
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subject to 


To the base problem we add the following algebraic equality constraints: 
hily,u) =0, ¿=1,2,...,1 or h(yu)20; 0<t<t; 


where l is less than m, the dimension of u. From Section 4.3.3.2 (p. 103), the 
base problem is equivalent to the optimization of 


te 


=] F+A'(-y +g) dt 
0 


subject to the usual preconditions of weak continuity and constraint qualifica- 
tion. Thus, the current problem is to find the optimum of J subject to h = 0. 
Observe that the optimum of J already implies the satisfaction of state equa- 
tions. Then, according to the Lagrange Multiplier Rule, the current problem 
is equivalent to finding the optimum of the further-augmented functional 


te te 


i 
Mu) = JG.) | 3 uti ato i [na 
p, l 0 
where y denotes the state vector that satisfies state equations for any ad- 
missible control vector u and v is the vector of time dependent Lagrange 


multipliers 


vt w(t) .. wt) 


The additional preconditions for the optimum are as follows: 


1. The partial derivatives of h are continuous with respect to y and u in 
the vicinity of the optimum at (y, à). 


2. The following constraint qualification is satisfied: there exists a set of 


m control variations (óu;,óus,...,óu4,) for which at each t in (0, tg] 
Shu (2; Our) Sha (2; dug) — óhi(2; fum) 
óh3(2; Our) óha3(2; dug) sae dh2(Z; dum) 
no = =0 
Ohm(Z;6U1) dhm(Z; u2) ... dhm(Z;dUm) 


where Z is the coordinate set (y, à) at the optimum. 
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Example 4.5 
In the last example, let us say we want uz to be increasingly higher for a 
smaller amount of the catalyst-to-reactant ratio. For this purpose we enforce 
the following equality constraint at all times: 

La 1 


=a —1 
Li pua—=c 


where a, b, and c are some suitable parameters. The objective is to minimize 
I subject to state equations and the above algebraic constraint. In turn, 
the equivalent objective is to minimize J given by Equation (4.15) on p. 105 
subject to the algebraic constraint. Provided that the preconditions for the 
minimum of J are satisfied, the objective is to find the minimum of 


tr 


1 
M=J+ fu ES a — -1 dt (4.16) 
qi puz c 
0 —— 
h 


where p is another time dependent Lagrange multiplier. 
The additional preconditions are as follows: 


1. The partial derivatives of h are continuous with respect to x and u in 
the vicinity of the minimum at (X, à). 


2. The following constraint qualification is satisfied at each t in (0,tr]. 
There exists a control variation du for which dh(x, à; du) = 0. 


[ 


4.4 Lagrange Multiplier and Objective Functional 


In most problems, a Lagrange multiplier can be shown to be related to the 
rate of change of the optimal objective functional with respect to the con- 
straint value. This is an important result, which will be utilized in developing 
the necessary conditions for optimal control problems having inequality con- 
straints. 

For simplicity, consider an objective functional J dependent on a control 
function u(t) and subject to the constraint 


K (u) = ko 


We assume that both J and K are Gáteaux differentiable. This is a modest 
assumption, which is valid in most optimal control problems we encounter. 
The reason for this assumption is the need for the linearity of the differen- 
tials in the following three-step derivation of the relation between a Lagrange 
multiplier y and the objective functional J: 
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Step 1 Let J be optimal at à, which depends on ko, the value of the con- 
straint. Then at any t in the t-interval, Taylor's first order expansion gives 


Q (ko + Ako) = û(ko) + the, Ako + € 
where e/Ako vanishes with Ako. We can rewrite the above equation as 


û(ko + Ako) = û(ko) + Ako (tik + e/Ako) 
du 


Step 2 Using 6u as indicated in the above equation, 


Jlú(ko + Ako)] = Jlú(ko) + Akoðu] 
= Jlú(ko)] + dJ[ú(ko); Akoðu] + ea (Ako6u) 
where dJ is the Gâteaux differential of J [from Equation (2.8), p.30] and 
€1/Ako vanishes with Ako. Since dJ is linear with respect to the second 
argument, i.e., Ako6y, we obtain, upon expanding du, 
J[û(ko + Ako]) = J[û(ko)] + dJ[ú(ko); Ûk Ako] + dJ[ú(ko); e] + €1(Ako6u) 
= Jlú(ko)) S AkodJ [i (ko); tik | + AkodJ [ü(Kko); €/Ako] + €1 


The above equation can be rearranged as 


J[ú(ko + Ako)] — J[ú(ko)] 


ae = dJ[f (Ko); no] +4 [ü(Ko); €/ Ako] + AE (4.17) 


Ak 
In the limit, Ako > 0, we have 


Jlú(ko + Ako)] — J [ü(ko)] 
Ako 
dJ[ú(ko);e/Ako] = dJ[ú(ko); 0] = 0 
€1/Ako =0 


= Ju (4) (i.e. the partial derivative) 


so that Equation (4.17) becomes 
Theo (a) = dJ[ú(ko); Ûko] 


We already know that if J is Gâteaux differentiable, then its variation 6J 
exists and is equal to dJ. Therefore, 


9Jlú(ko); tiko| = Jko (à) 


Lagrange Multipliers 109 
Step 3 The augmented objective functional is M = J + uK where p is a 
Lagrange multiplier. From the Lagrange Multiplier Theorem, assuming that 
OK [ü(ko); úx,] = 0, we have 

9M [ü(Ko); ûko] = 9J [ü(Ko); no] + uK [Q(Ko); o] = 0 

'The last two equations yield 

Ji, (&) = —póK [ú(ko); Uno] (4.18) 
Repeating Step 2 for K in place of J, we obtain 

SK [â(ko); à,] = Kro (0) 
Expanding the right-hand side of the above equation, 


7 . ko + Ako) — k 
8K[à(ko); in, = ROS 


=1 (4.19) 
From Equations (4.18) and (4.19), we finally obtain 


(4.20) 


4.4.1 General Relation 


The above result can be readily generalized for the optimal control problem 
in which J is dependent on vectors y and u of state and control functions 
and is subject to m constraints, K; = ki, i = 1,2,...,m. In this case, the 
Lagrange multipliers are given by 


9 
pi > — DIDA); ¿=1,2...,m (4.21) 
T 
where k = kı k2 dla Kg . 


4.5 John Multiplier Theorem for Inequality Constraints 


In this section, we will derive the John Multiplier Theorem, which is a set of 
necessary conditions for the minimum of an objective functional constrained 
by inequalities. 

We begin with an objective functional J dependent on a control function 
u(t) and subject to the constraint K(u) < ko. As before, we assume that both 
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J and K are Gáteaux differentiable since we will need the continuity of the 
differentials. 

Let J be minimum at & among all u satisfying the inequality constraint. 
Then K(ú) could be either ko or less. We will consider these two cases as 
follows: 


Case 1 Here K(ú) = ko and the inequality constraint is said to be active. 
The augmented objective functional is M = J + uK where y is a Lagrange 
multiplier. The Lagrange Multiplier Theorem yields 


OM =J + pok =0 
where, from Equation (4.20), 


p= -z Tato) 


Now, any change in K from K(&) has to be negative since K(u) € ko. This 
change, which is Ako, cannot decrease the value of J because J(ă) is already 
the minimum subject to the constraint K(u) € ko. Hence, the change AJ has 
to be positive or zero, corresponding to the negative Akg. In other words, the 
partial derivative in Equation (4.20) has to be either negative or zero. This 
finally means that 


p>0 


Case 2 Here the strict inequality K(û) < ko is in effect and the inequality 
constraint is said to be inactive. Let K(ú) = ke where ke is some real number 
less than ko. Then Equation (4.20) for K(à) = ke is 


0 


5x, eE) 


HRS 


The continuity of K implies that there is an interval around ke in which 
K < ko for all u in a region around å. In that interval, Ak, can be positive or 
negative, but AJ has to be positive or zero since J(&) is already the minimum. 
Thus, the partial derivative in the above equation can only be zero, thereby 
leading to 


p=0 
In this case, the Lagrange Multiplier Theorem yields 6M = 0, which is the 
necessary condition for the minimum. 


The John Multiplier Theorem can now be expressed by combining the re- 
sults for both the cases as follows: 
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The necessary condition for the minimum at &(t) is 


5M (å) = 8J(à) + nóK (à) = 0 
K(à) < ko 
p> 0, p[K (a) — ko] = 0 


— MM! 
complementary slackness condition 


The following preconditions must be satisfied: 


1. The Gáteaux differentials of both J and K are weakly continuous near 
a(t). 

2. The constraint qualification — There exists a du for which K (i; du) = 0 
whenever the constraint is active. 


Observe that the complementary slackness condition, p[K(ú) — ko] = 0, 
requires u to be zero when the constraint is inactive. Otherwise, u could be 
Zero or greater. 


Example 4.6 
Let the problem in Example 4.2 (p. 102) be constrained by the inequality 


u<cy, at t=t 


where c is some constant. Thus, the minimum of the augmented functional J 
given by Equation (4.14) is now subject to the inequality constraint 


u—cy < 0 at t = tf 
= = bos a 
K ko 
We assume that the preconditions already hold for the minimum of J. 
Now the minimum of J implies that the state equation is already satisfied 
for the given initial condition y(0) = 0. The augmented functional is given by 


MG u) =I, u) c uK 2 Jt n u-ci, 


=tf 


where ĝ is the state satisfying the state equation for any admissible control 
u and u is an undetermined multiplier. Hence, from the John Multiplier 
Theorem, the necessary conditions for the minimum of M($, u) at ú are 


6M(j,à) = 0 
ú=cy < 0 at tr 
p 2 0, u(â — cj) at tr 
—— ————— 


complementary slackness condition 


subject to the following additional preconditions: 
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1. The Gâteaux differential dK (f, u; du) is weakly continuous near ú for 
t = te. This means that the partial derivative K, at Y is continuous. 


2. The constraint qualification — If the constraint is active, there exists a 
control variation du for which ôK (y, û; 6u) = 0 for t = tf. It means that 
Ky, — 0 at (9, à) for t = tr. 


Example 4.7 
Let us replace the point inequality constraint in the above example by 


K=u-cy € 0, for all t in (0, t] 
In this case, with the values of t defined as 
to =0, ty = to At, ta = ti + At, "m tn—1 = tn-2 + At, tn = te 


with At tending to zero, the inequality constraint can be rendered in terms 
of the following inequalities: 


u(t;) — cy(ti) € 0; i—0,L...,(n— oo) 


Hence from the serial application of the John Multiplier Theorem, [similar to 
that in Section 4.3.3.1 (p. 100)], the final augmented functional is given by 


noo te 
M(j,u) — J(j,u) + ÎL mKi(Bi ui) = + J hu — cg) dt 
i=0 0 
where the multiplier y is a function of t and the necessary conditions for the 
minimum are 


te 


8M (j, à) = 8J(g, à) + J uà K (6,4) dt = 0 


p>0, u(à — cj) =0 for all t in [0, ts] 


complementary slackness condition 


'The additional preconditions of Example 4.6 now apply to each t in the interval 


[0, ts]. [] 
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4.5.1 Generalized John Multiplier Theorem 


The approach in the preceding section may be followed to arrive at the fol- 
lowing the John Multiplier Theorem for several inequality constraints: 

The necessary conditions for the minimum of J (y, u) subject to K(y,u) € k 
at ú(t) are 


SMG, å) = 5J($,)-- n 5K(9,4) = 0 


A 

e 
r È 
IV JA 
o 


T ó i "ES 

u [K(y,à) -k = 0 

NÁ o 
complementary slackness condition 


where M = J + p'K. The the following preconditions must be satisfied at 
the minimum: 


1. The Gáteaux differentials of J and K are weakly continuous near ú. 


2. The constraint qualification — Whenever la of the total | constraints 


are active, there exists a set of control variations (Óui, dug, ..., 9uy,) 
for which 
óK1(2;óui) óK1(2;óua) aa ó K1(2; Ou, ) 
óK»(2;óui) óK»(2;óua) ons ó K»(2; Ou, ) 
mo = l | | =0 
Kı, (2; óu1) ó K;, (2; 6u2) ... SK, (2; du, ) 


where Z = (y, ú) and each 0K,(2;9u,) corresponds to an active con- 
straint K;(y,u) = ki. 


Example 4.8 
Let us restrict the second control 
Umin < U2 < Umax 
in the problem of Example 4.5 (p. 107). Thus, the problem is to minimize 


te o 


T= y (xi a2)" + (u; —u®)? dt 
0 i=l 


subject to 
y U1 
X1 = y (ae = zı) = kzz, xı (0) = X1,0 
: uo — UE 
T2 = A a, x2(0) = 220 


V 
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and the following algebraic equality and inequality constraints: 


x 1 

2 =a —1 

X1 pua—=c 
—U2 + Umin < 0 


U2 — Umax < 0 


The equivalent problem is to minimize J given by Equation (4.15) on p. 105 
subject to the initial conditions and the algebraic constraints. It is assumed 
that the preconditions for the minimum of J (see p. 105) are satisfied. The 
augmented functional for this modified problem is given by 


qi 1 
m=3+ f Hı ane iue nC tin) "p Uma) at 
0 —— — Ko K3 
Ki 


where 41, 12, and ug are the time dependent Lagrange multipliers. 
From the Generalized John Multiplier theorem, the necessary conditions 
for the minimum are 6M = 0 and 


— U2 € "min H2 2 0 pa (—ua + Umin) =0 
U2 < Umax H3 > 0 [3 (uo == Umax) =0 


throughout the time interval [0, tg] subject to the following additional precon- 
ditions: 


1. The Gâteaux differentials of Kı, K2, and K3 are weakly continuous at 
X near ú. In other words, the partial derivatives OK,/0u; (i,j = 1,2) 
at X are continuous. 


2. The constraint qualification — 


2.a The first constraint is always active, for which Kı = 0. If only the 
first constraint is active, then there exists a control variation óu 
for which ôK; (£, G; du) = 0. It means that 


T 


Qui Qu» £,ü 

2.b At any time, if the first and any of the remaining two constraints 
are active, then the corresponding vectors of partial derivatives, 
Kya and (Koa or K3q), are linearly independent. 


Note that the last two constraints can never be active together. 
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4.5.2 Remark on Numerical Solutions 


When solving an inequality-constrained optimal control problem numerically, 
it is impossible to determine which constraints are active. The reason is one 
cannot obtain a u exactly equal to zero. This difficulty is surmounted by 
considering a constraint to be active if the corresponding u € a where a is a 
small positive number such as 107* or less, depending on the problem. Slack 
variables may be used to convert inequalities into equalities and utilize the 
Lagrange Multiplier Rule. 

Alternatively, increasing penalties may be applied on constraint violations 
during repeated applications of any computational algorithm used for uncon- 
strained problems. We will use the latter approach in Chapter 7 to solve 
optimal control problems constrained by (in)equalities. 


Note 


When using Lagrange multipliers in the rest of the book, we will skip men- 
tioning the preconditions assuming that they are satisfied. 


4.A Inverse Function Theorem 


Let f be a continuously differentiable function of x, both having the same 
dimension n greater than zero. If the derivative f'(x) is non-zero at x — xo 
for which yo = f(xo), then there exists a continuous inverse function fin” (y), 
which maps an open set Y containing yo to an open set X containing xo. 
Note that an open set has each member completely surrounded by members 
of the same set (see Section 9.3, p. 268). Moreover, the inverse function is 
differentiable, the proof of which can be found in Rudin (1976). 


Remark 


Under the given conditions, the theorem assures that y — f(x), which is a set 
of n equations 


Yi = fi(21,22,..., Un), 1—1,2,...,n 


can be uniquely solved as x = fi" (y), i.e., 


mi = FE ondes). 1—1,2,...,n 


in sufficiently small neighborhoods of xy and yo. 
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Outline of Proof 


Based on the given function f, we will develop an auxiliary function g. We 
will show it to be a contraction, which is associated to a unique fixed point. 
This property will then lead to the existence of the inverse function finY. We 
start with the description of a contraction and its fixed point. 


Contraction 


A contraction is defined to be a function @ that maps a region X to itself 
such that for all x; and x; in X 


le) — é(x;)ll < elx: = xil; c«1 (4.22) 


In the next three steps, we will show that there is a unique fixed point x in 
X such that x — p(x). 


Step 1 Let us select an arbitrary xo in X and obtain the series 
Xi41 = O(x;); i20,,2,.. (4.23) 
Then for i > 0 
[ia xil = Bi ia] x clxi—xi-a ll 


where $; = P(x;) and the above inequality follows from the definition of a 
contraction, i.e., Inequality (4.22). Its recursive application yields 


laca — xil < elxi—xiil < cHxii-x-sl < 
c!||x1 — xol| (4.24) 


IA 


Step 2 For j >i, we can write 


[xj — xl] = lies — x5) + (xi+2 — Xi41) + +++ (a — Xj2) + (x; — x;i-)ll 


IIrll 


Applying the triangle inequality (Section 2.2.2, p.26) on the right-hand side, 


ri < llos — xil + [loci — 6411] +--+ + lx 5-1 — xj-2ll + lxi x- 


Combining the last two results, 


[xi — xjll S [Xi — xil + [Riza — xil: [xo 5-1 — Xa + lx; x- ll 
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Step 3 Using Inequality (4.24) for each right-hand side term above, we get 


Ii = xyli € (ei HEH eee? ec la — xe 
M ^ 


P 


Because c is a positive fraction, the coefficient on the right-hand side 


PEXPÓTÓU Y. (lete...) pam 


because of which 


Fixed Point 


In the above inequality, with c < 1 and i tending to infinity, ||x; — x;|| tends 
to zero and so does $; — 6; due to Inequality (4.22). In other words, x; 
and q, tend respectively to some x and [= $(x)] in the region X. Hence, 
Equation (4.23), i. e., 


lim X41 = lim $i 
1—00 1—00 
is equivalent to 
x = (x) 


where x is the fixed point. 


Uniqueness of the Fixed Point 


The fixed point is unique because otherwise if there is another fixed point, 
say, X = (x), then ||@(x) — (xX)|| = ||x — X||, which contradicts Inequal- 
ity (4.22). 

Having shown that a contraction has a unique fixed point, we consider 
the givens of the Inverse Function Theorem. Based on f(x), we propose an 
auxiliary function and prove it to be a contraction. 


Auxiliary Function 


Consider the auxiliary function 


g(x) == (4.25) 


where y is an n-dimensional vector, D = f'(xo) and x is in the vicinity of xo 
such that ||x — xo|| < 6 for some 6 > 0. In the next four steps, we show that 
g(x) is a contraction. 
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Step 1 The continuity of f' at xo implies that there exists an open set X in 
which 
lx- x9||< 6, $>0 (4.26) 


such that 
lf’ (x) -Dl «e, a>0 (4.27) 


Differentiating g(x) with respect to x, we get 
g'(x) = I- DP (x) = D![D — f'(3) 


where I is the identity matrix. Taking the norm on both sides of the above 
equation and applying Inequality (4.27), 


lg'G)l = D |D- f'G9| 
« D^! €1 


With the choice e = 0.5/ D^! , we have 


|g (x)]] < 0.5 (4.28) 


Step 2 Considering x as 
x(s =(1—s)xo+sx1, 0<s<l (4.29) 
we obtain g(s) — g[x(s)] so that 


dg aa , 
E = g'(x) = =8 (960 —x0) 


Taking the norm on both sides of the above equation, we get 


Applying the Operator Inequality (Section 9.24, p. 281) on the right-hand 
side of the above equation, 


IIs’ (x) (xx — Xo) || < Ile’ Gol] (xa — xo) 
Combining the last two results, we get 


dg 
ros Ile’ (x) ll: — xoll 
S 


Using Inequality (4.28) in the above result, we obtain 


d 
SE < 05|x; — xo] (4.30) 
ds 
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Step 3 Let h(s) = [g(1) — g(0)]'g(s). Then for 0 € s € 1, the Mean Value 
Theorem for derivatives (Section 9.14.1, p. 276) yields 
dh - dg 


^()-A()- LE Q-0)-lg()- a0)" E 


Also, from the definition of A(s), 


h(1) — ^(0) = [8(1) — 8(0)]' [8(1) — &(0)] = llg(1) — &(0)I]" 
From the last two equations, 


lg) — &0)I^ = lg(1) — &(0)]" = (4.31) 


Applying the Cauchy-Schwarz Inequality (Section 9.23, p. 281) to the right- 
hand side of the above equation, we get 
dg 
< lle) - 80) 3; 
E 


S S 


Using the above inequality in Equation (4.31), we get 


llg(1) — g0)? < llg(1) — ss) z 
or 
dg 


le()-sOl< 3 


Step 4 From Equation (4.29) 
g(1) = g[x(1)] = g(xı) and g(0) = glx(0)| = g(xo) 
Therefore, the last inequality can be written as 
dg 
ds 


S 


[g(x1) — g(Xo)|| < 


Applying Inequality (4.30) in the above result, we get 


ligo) — g(xo)l] < 0.5[|xx — xo] 


Hence, g(x) is a contraction as defined by Inequality (4.22). Being a contrac- 
tion, it has a unique fixed point. 


Existence of the Inverse Function 


We will now use g(x) to show that f(x) is one-to-one (injective) as well as onto 
(surjective) over regions X and Y, which are open sets. Note that this state- 
ment is equivalent to the fact that there exists a continuous inverse function 
f(y) that maps Y to X. 
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Injection of f(x) 


Consider g(x) defined by Equation (4.25) in the open set described by In- 
equality (4.26). Since g(x) is a contraction, it is associated with a unique 
fixed point given by x = g(x) or 


fx)—y 
x=x D 
Since the determinant of D is not zero, further simplification yields 
f(x) = y 


which holds for exactly one x in X. Hence, f(x) is injective in X. 


Surjection of f(x) 


Let Y be the collection of points y = f(x) for all points x in the open set X. 
Then obviously f(x) is a surjection from X to Y. 

That Y is an open set follows from the continuity of f(x). It means that 
each point y; = f(x;) in Y lies in an open set surrounded by neighboring 
points y = f(x). In other words, the open set ||y — y;|| < € corresponds to 
an open set |x — x;|| < 6 in X where both e and 6 are some positive real 
numbers. Thus, Y, being a collection of open sets, is an open set. 
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Exercises 


4.1 Show that the costate variables in the optimal control problem 
a. are continuous with respect to time, and 
b. have piecewise continuous time derivatives 


4.2 Show that the augmented functional formed by adjoining even the initial 
condition of the state equation leads to the same necessary conditions for the 
optimum. 


4.3 Expand the determinant mo in Example 4.4 (p. 104). 


4.4 Simplify the constraint qualifications in Section 4.3.3.3 (p. 105) and Sec- 
tion 4.5.1 (p. 113). 


4.5 Derive the necessary conditions for the minimum of the batch distillation 
problem described in Section 1.3.1 (p. 5) without the purity specification. List 
all involved assumptions. 


4.6 Repeat Problem 4.5 including the purity specification. 


4.7 Find the necessary conditions for the maximum in Example 2.10 (p. 45) 
in presence of the following inequality constraints: 


T < Taso 0O<t< tr 
u(te) < oe 


State all assumptions involved. 
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Chapter 5 


Pontryagin's Minimum Principle 


One of the most profound results of applied mathematics, Pontryagin's 
minimum principle provides the necessary conditions for the minimum of an 
optimal control problem. The elegance of the principle lies in the simplicity of 
its application to a vast variety of optimal control problems. Boltyanskii et al. 
(1956) developed the principle originally as a maximum principle requiring the 
Hamiltonian to be maximized at the minimum. 

In this chapter, we will present the proof of the minimum principle. The 
minimum principle uses a positive multiplier for the objective functional in 
the Hamiltonian formulation.* With this provision, the minimum principle 
concludes that the minimum of the problem requires minimization of the 
Hamiltonian in an optimal control problem whose minimum needs to be de- 
termined. 

Some readers may first want to get the essence of the minimum principle and 
go cursorily over the derivation in Section 5.4. This section may be skipped 
during the initial reading. 


5.1 Application 


Before delving into the proof, let us take the simplest optimal control problem 
and examine the application of Pontryagin's minimum principle. We will 
realize that we already have been applying the minimum principle to our 
optimal control problems. 

Consider the minimization of 


I= | Fly, u) dt (3.4) 
| 


subject to G(y,g,u) = —y + g(y,u) - 0 (3.5) 


* A negative multiplier, on the other hand, leads to the maximum principle, which stipulates 
that the Hamiltonian be mazimized in a minimum problem. 
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with the initial condition 
y(0) = Yo (3.6) 


This problem is the same as the one in Section 3.2 (p.58). The Hamiltonian 
for this problem is defined as 


H(y,A,u) = F(y,u) + Ag(y, u) 
According to Pontryagin's minimum principle, if à is optimal, then the cor- 


responding Hamiltonian H($,A,à) at each time instant is minimum over all 
admissible choices for u. Thus, if u = û + du is any admissible control, then 


(5.1) 


Let us understand the import of the above result in light of the optimal control 
analysis we have done so far. The augmented functional for this problem is 


f= J Piya) PCW EAE (3.7) 
0 


whose variation is given by (see Section 3.2.2, p. 59) 


te 


tr 
TE Jr e dt + [cir 
0 


te te te 
" t 
= Je, +AG, + ÁJóydt— Ay Ñ «fe +AG,)budt-+ f Gat 
0 


0 0 

te te te 

: t 

= Ja, + AÀ)óydt — Ady i + [ nonae f (y+ món) dt 

0 0 0 

If, for a given control a(t), we obtain 
1. the state y(t) that satisfies y = Hx or G = 0 with $(0) fixed to yo, and 
2. the corresponding costate A(t) that satisfies A = —H, with A(tf) fixed 

to 0, 
then we are left with 


te 
8J = Juana 
0 


where, since G = 0 is satisfied, 


u 
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for sufficiently small variation du in ú. 
Now, for sufficiently du at any time, we can express the Hamiltonian at 
(9, A, à + du) using the first order Taylor expansion as 


H(GĂ, @+ du) = H(j, Â, à) + Hu(9, A, adu 


where u is the perturbed control (à + ĝu). From the last two equations, we 
obtain 


te tf 


16,0) - 163) = [ Hali A uat = [iii X) - HO) dt (5.2) 
0 0 


Depending on the control u, the following cases arise for the above equation: 


Case 1 There is no constraint on u. Then du can be positive or negative at 
any time instant. Now a non-zero H., (4, A, ù) at any time allows the possibility 
of I(j,u) — I(9,4) < 0. Therefore, H.(%, A, â) must be zero to ensure the 
minimum of J. This necessary condition, as observed from Equation (5.2), is 
equivalent to . 

H(9,A,u) = H(j,À, à, 0<t<tr 


which is included in the Pontryagin’s minimum principle, i. e., Equation (5.1). 


Case 2 The control u is in between (but not at) its specified upper and 
lower limits, Umax and Umin, respectively. Each time when that happens, the 
necessary condition for the minimum of / is 


H(9,A,u) = H(g, À, à), 


for the same reason as in the previous case. 


Case 3 The control is constrained by u < Umax. If at any time @ = Umax, 
then du can only be negative. A positive óu will make the control exceed 
Umax and therefore be inadmissible. Hence in this case, H.,(4, A. à) must be 
zero or negative to ensure the minimum of J, i.e., I(g,u) — I(g, à) > 0. This 
necessary condition, as observed from Equation (5.2), is equivalent to 


H(, À,u) > H(G, A, à), 


as asserted by Pontryagin's minimum principle, i. e., Equation (5.1). 


Case 4 The control is constrained by u > Umin. If at any time i = Umin, 
then du can only be positive. A negative 6u will make the control less than 
Umin and therefore be inadmissible. Hence in this case, H.,(4, Fa à) must be 
zero or positive to ensure the minimum of J, i.e., I(j, u) — I(g, à) > 0. Thus, 
the necessary condition is the same as in the previous case. 
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Case 5 The integrand F in Equation (3.4) is a function of |u|. Now, when 
u = 0 the partial derivative of F,, does not exist, and we cannot apply the 
stationarity condition, H, = 0. However, Pontryagin's minimum principle 
does not require the partial derivatives Fi, and gu to exist. According to the 
principle, , " 

H(9,5,u) > H(j,À, d) 
is the necessary condition for the minimum of J in this case as well. 

Freedom from having the partial derivatives of F and g with respect to 
u means that the principle is also applicable to the case when only a finite 
number of controls are available. 

The above cases show that Pontryagin's minimum principle provides an 
overarching necessary condition for the minimum. Appreciating this fact, we 
present a general optimal control problem involving a wide class of controls 
for which we will derive Pontryagin's minimum principle. 


5.2 Problem Statement 


It is desired to minimize the objective functional 
te 
T= | Fist) n0... (t), un) ua... ( d 
0 


where t or time is the independent variable, u;(t)s are the controls, and y;(t)s 
are the state variables governed by the following ordinary differential equa- 
tions: 

dy; : 

ES gily1(t), ya(t),... ,ynlt), ux (t), u2(t),..., wt); ¿1=1,2,...,n 
called the state equations. These equations are autonomous because no g; 
depends on the independent variable t. The objective functional I is also 
autonomous since F' does not depend explicitly on t. 

The initial conditions for the state equations are 


yi(0)—yio; ¿=1,2,...,n 


While the initial time is fixed at zero, the final time tf is not specified or fixed. 


5.2.1 Class of Controls 


For the above problem, we consider a general class of piecewise constrained 
controls that are typically encountered in practice. The controls in this class 
have the following characteristics: 
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1. The controls u;s are piecewise continuous with respect to t. Figure 3.4 
(p.77) showed one such control, which is made of three continuous 
curves. When two such curves meet, there is a jump discontinuity, e. g., 
at time tı, as shown in the figure. On either side of tı, the control is 
provided by the curve on that side. At t > t4 we take the control value 
from the right-hand side curve. 


Note that the number of jump continuities is finite.* In other words, 
each curve spans a non-zero time duration. Obviously, a single contin- 
uous curve for the control is also a member of this class of controls. 


2. The controls are not discontinuous at the initial and final times. 


Thus, the control vector u(t) approaches u(0) as t tends to 0 from the 
right-hand side. Similarly, u(t) approaches u(tf) as t tends to tf from 
the left-hand side. 


3. At all times, each control takes values from a bounded set of values. 
For example, a control u; may be specified to take values from the 
constrained set (0 < u; < 5}. 


5.2.2 New State Variable 
We also introduce go = F and a new state variable yo defined by 


= = goly1(t), ya(t), sad ; Va (t), ul (t), ua(t), ees Um (t)); yo(0) = 00 


Thus, yo(tf) is equal to J, as can be verified by integrating the above dif- 
ferential equation. Next, we define the Hamiltonian, 


H= Ao(t)go ZE A(t)gi ón An(t)9n 


where A;s are the time dependent Lagrange multipliers or costate variables. 
This is the same definition as Equation (3.34) on p. 68 with Ag = 1. 


5.2.3 Notation 
For the sake of convenience, we adopt the following notation: 


1. To reduce clutter, we omit (t) unless needed for clarity. The time- 
dependence of yis, u;s, and As is taken for granted. 


2. The vectors y, A, and g are, respectively, 


Yo Ui e Yn» Ao ^ .. An and 


go gı ... Jn 


* Otherwise, with infinite jump discontinuities, the control would never be continuous. 
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3. The i-th component of g is the function 
gi(y,u) = gi(Y1,Y2,---,Yn; U1, U2,---,Um) 
It is implicitly understood that any g; does not depend on yo. 
4. The costate at the minimum is denoted by A without the hat ”. 


Using the above notation, the state equations can be written as 


. dy | 
y = q, = 8(y,u) (5.3) 
and the Hamiltonian can be expressed as 
H(y,A,u) = A g(y,u) (5-4) 


From the last two equations, we get the identity 
OH dy; OH 


pao ells, . 
Y dt = Oh i=0 n (5.5) 


5.3 Pontryagin's Minimum Principle 


Pontryagin’s minimum principle is a statement of necessary conditions for 
the control to be optimal. The main conclusion of this principle is that the 
optimal control minimizes the Hamiltonian at each point in the time interval 
whether or not the control there is continuous. 


5.3.1 Assumptions 
The principle is based on the following assumptions: 


1. The controls belong to the class of piecewise continuous controls defined 
in Section 5.2.1. 


2. The functions 


for i, j —0,1,...,n 
Oyj 


Ji (y, u) and 
are continuous in the space of functions 


(Y1, Y2; +++) Yn, U) 
where u takes values from the m-dimensional constrained space includ- 
ing all boundary points. 


For example, if 1 < uj < 2, then the functions are required to be 
continuous at u = 2 even though u cannot be equal to 2. 
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5.3.2 Statement 


If ú is the optimal control resulting in the optimal state y over the time 
interval [0, te] in which J is minimum, then there exists a non-zero vector A 
such that the following conclusions hold: 


1. The corresponding Hamiltonian 


H 


Il 
A 
< 
> 
e 


is minimum over the set of all admissible controls at each point in [0, te] 
whether or not the controls are continuous at the point. To be specific, 


H < H, A, u) 
at each t in [0, ty]. 


2. Both Ê and Xo are time invariant, i.e., constant over [0, te]. 
3. If the final time is not fixed, then H = 0 at the final time ĉr. 


Remarks 


1. The first conclusion means that throughout the time interval, the opti- 
mal control à minimizes the Hamiltonian H($, A, u). 


2. The last two conclusions together imply that if the final time is not 
fixed, then 


x 
E 


throughout the time interval. 


5.4 Derivation of Pontryagin's Minimum Principle 


Very versatile in applications, Pontryagin's minimum principle is among the 
most profound and difficult results to derive. Figure 5.1 presents the outline 
of the derivation of the principle. 

Basically, we intend to perturb the optimal control along the control axis 
as well as change the final time to examine the effect on the Hamiltonian. 
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consider perturbations in the 


optimal control ú by 


1. introducing a pulse at a time 
when ú is continuous 


(Pulse Perturbation) 


2. changing the final time 
(Temporal Perturbation) 


given a pulse perturbation 


V 


show that the dot product of A 


and the state change is constant 


V 


Y 


get first order accurate changes in 


final states for both perturbations 


show that ú minimizes the 
Hamiltonian H = H(y, A, u) when 


ú is continuous 


| 


show that final states are convex 


and supported by a hyperplane 


l 


select 


1. the costate A as adjoint to 
the change in the final state 


2. A at the final time opposite to 


the normal to the hyperplane 


show that 


1. in a subinterval where û is 
continuous, the minimum of 
H or H($, A, à) is 

a. constant in a subinterval 


where ü is continuous 


b. the infimum of H($, A, u) 


w.r.t. the value of u 


2. the infimum is continuous 
throughout the entire 


time interval 


y 


y 


given a temporal show that Ao 


perturbation is a constant 


conclude that the Hamiltonian 
corresponding to à, i.e., H($, A, à), 


is zero at the final time 


conclude that H($, A, à) is 


1. defined even when ú is not 
continuous (i. e., à minimizes 
H even when ú is not 


continuous) 


2. constant throughout the 


entire time interval 


Figure 5.1 Outline of the proof of Pontryagin's minimum principle 


Figure 5.2 shows these two types of perturbations. While the first type 
involves a pulse perturbation in a finite subinterval, the second one has 
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A Bani ic A niz dă ali i 


perturbed îi; 
pulse 


perturbation 


a; is continuous temporal 
at tı perturbation 


rib: 0 (tr) 
Tw na 
to la i 


Figure 5.2 Two types of perturbations in à;, the i-th component of the optimal 
control û 


temporal perturbation in the final time. These perturbations change the 
optimal state generated by the optimal control. 

The time intervals of both perturbations are small enough to ensure the 
accuracy of the first-order approximation of the resulting new state. We will 
now examine these perturbations in the optimal control and determine the 
resulting states. 


5.4.1 Pulse Perturbation of Optimal Control 


Consider a subinterval 
[ty — Atı, tı), i.e., tı — Atı <t<t 


where the optimal control ă(t) is continuous at ti. In this subinterval, we 
consider a perturbed control u(t) to be constant, i.e., a vector v of some 
constants within the specified control limits. Elsewhere in the time interval 
0, î], u is the same as the optimal control ú. Figure 5.2 shows a pulse 
perturbation in the i-th component of à. 

In the next three steps, we determine the final state y(ts) due to a pulse 
perturbation in ú at time ti. 


Step 1 Integrating the state equation over the subinterval [tı — Atı, t1) for 
the controls u and ú, we get, respectively, 


ty tı 
tı 


Yia = f ewwa ad  $haQ- [ sa 


tı—âtı tı—Atı 
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Since there is no perturbation prior to (tı — Ata), y at this time is the same 
as y. Hence, the difference between the above two equations gives 


ti 


yu) 5) f gv.) - a] at (5.6) 
t1— At; 
Step 2 Being a continuous function of (yo, y1,..., Yn) as per Assumption 1 


(Section 5.3.1, p. 128), g(y, v) approaches g(y,v) as y tends to y in the 
subinterval [tı — Atı, ti). Thus, we can write 


[gi y; v) 7 gi(y, v)| 
ly — y 
where c;s are some positive constants,* and y is sufficiently close to y. Simi- 
larly, since y approaches y as the perturbation interval At; tends to zero, 


lly — yl 
———— E 
Ay 


where co is some positive constant, and At, is sufficiently small. Multiplying 
together the last two inequalities, 


lim lg (y, v) — gil¥, v) < d¡At1; 1=0,1,...,n 
At1—0 


where each d; = coc; is a non-negative constant. The above inequality is 
symbolically expressed as 


gily, v) — g:i, v) = O(Ati); ¿=0,1,...,n 


using the big-O notation explained in Section 9.5.1 (p. 269). Here O(At4) de- 
notes the maximum of the absolute error |gi(y, v) — gi(y, v)|. This maximum 
is equal to some positive constant times Atı when At is sufficiently small. 
In vector form, 


8(y, v) = g($. v) + O(At1) 
Using similar reasoning as above, we obtain 
g[$ (t). v] = gl$ (t), v] + O(At,) 
gii (t), à(t)] = g[$ (t1), ú(t1)] + O(At) 
where t lies in the subinterval [tı — Atı, tı). The last three equations combined 


with Equation (5.6) provide the new state due to the pulse perturbation at 
time tı as follows: 


* Since the left-hand side of the inequality cannot be negative, c;s cannot be zero or negative. 
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y(t1) = y (t1) + J 8[9(t1), v] — ely(t1), û(t1)] + O(Atı) dt 
ti—Ati 
= Şt) + ely(t1), v] — 8el9(t1), it) Ah +  O(At) (5.7) 


— mu 
N 
second order term 
first order term 


We consider At; to be sufficiently small so that the second order term vanishes. 
With this provision, the change of state at tı is 


Ay(t1) = y (t1) - $(t1) = ely(t1), v] — ely(t1), 4(t1)] Atı (5.8) 


Step 3 From time tı to the final time fr, the control is optimal, i.e., à, and 
the state equation is y = g(y,ú). Replacing y by (3 + Ay) and using the 
first order Taylor expansion for sufficiently small Ay, the state equation can 
be written as 


d, ` . uA . 
uM + Ay) = g(3 + Ay, à) = g(y, à) + gyAy, ti<t<ts (5.9) 


where gy is gy evaluated at (y, à). Since dy/dt is g(y, à), Equation (5.9) 
simplifies to 
d 
dt 
which is a homogeneous linear differential equation with the initial condition 
given by Equation (5.8). Integrating the above equation, we get for t > t1 


Ay = gg Ay, ti <t< ty (5.10) 


Ay(t) = J “Ay dt = W(t) Dv !(t)Ay(t) (5.11) 


where W(t) is an (n + 1) x (n + 1) fundamental matrix whose columns are 
the linearly independent solutions of the differential equation, Equation (5.10) 
(see Section 9.26, p. 283). Observe that the above equation along with Equa- 
tion (5.8) reveals that Ay is directly proportional to Atı. We ensure the 
requisite size of Ay in Equation (5.9) by having Atı suitably small. 

Now the new state at time t > tı, utilizing Equation (5.11), is given by 


t t t 
| [dy fay d o, = 
y(t) = Zar | Pars | yat - st) wow (t3) Ay (&) 


The above equation, upon substituting for Ay (tı) given by Equation (5.8) and 
replacing t by tf, yields the final state in the presence of the pulse perturbation, 


y (61) = $ (£0) + VE) t) gl$(6).v] — &($ (65), à(6)] At 
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which is first order accurate with respect to Ati, the interval of pulse per- 
turbation. We take At; to be small enough for the above equation to be 
true. 


5.4.2 Temporal Perturbation of Optimal Control 


As shown in Figure 5.2, we now consider a truncation or extension of the 
optimal control à at the final time tf. If Af; is negative, then the control 
remains ú in the truncated interval [0, £c — |Ate]]. But if Atg is positive, then 
the control remains ú in the original interval [0, £j] and stays at the optimal 
value (ts) for t > tr in the added subinterval [fr, fe + Ati]. 

Integrating the state equations from f; to fp + Ate, we get 


ig Ate 
yi + Ate) = y (5 + J gly(t), à(£)) dt 


te 


For sufficiently small At, the integral in the above equation is given by its 
first order Taylor expansion, g[y(tr), O(t¢)| Ate. Thus, 


y(i + Ate) = y (te) + ely (tr), à(£)] At (5.12) 


is the first order approximation of the final state due to the temporal pertur- 
bation. Similar to At, above, we take At; to be small enough for the above 
equation to be true. 


5.4.3 Effect on Final State 


Consider a collection of final states generated by the optimal control altered by 
all possible combinations of pulse and temporal perturbations. This collection 
also includes the optimal final state generated by the optimal control without 
perturbations. Appendix 5.A (p.145) shows that this collection is a convex 
set. As shown in Figure 5.3, a convex set contains all points that lie on a 
straight line joining any two points of the set. 

In this set, let yf be a point representing a final state, which is either 


1. y (tg) due to pulse perturbation, or 
2. y(t; + Atr) due to temporal perturbation. 
Then, since the collection is convex, the final state given by 
zi—ayr--(1—o)y; for 0<a<l 


also belongs to the set where y; is another member of the set, i.e., another 
final state. 
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convex set S. 


z is any point 


/ on the line 


eeu 


hyperplane P 
yı 


boundary point y; 
0 > 7/2 


normal p — x — y, 


ya 


Figure 5.3 A convex set S of final states with a supporting hyperplane P. A 
hyperplane is a tangent line in two dimensions 


Supporting Hyperplane 


Now for a convex set, there exists a supporting hyperplane at any boundary 
point of the set (see Appendix 5.B, p. 149). We apply this result to our convex 
set of final states, in which the final optimal state 


- 
e= folte) tilts)  ... «o 


is a boundary point since it cannot be an interior point. Otherwise, y; would 
have been surrounded in all directions by other points of the set, including 
those with yo(tf) components less than g(t). This situation is contradictory 
since no point in the set can have the yo(ts) component — the objective 
function value 7 — less than $o(fr), which is the minimum J. 

In light of the above facts, there exists at y; a hyperplane supporting the 
convex set of final states. Let 

2 
P= Po pi +++ Pn 


be the normal to the hyperplane such that no point of p is in the set, as 
shown in Figure 5.3. Hence, for any point yr in the set, the angle between the 
vector (yr — ys) and the normal p is greater than or equal to 7/2. Derived in 
Appendix 5.B, this result is mathematically expressed by the inequality 

a 


p (yr- 0x0 
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5.4.4 Choice of Final Costate 
Let us choose A(ts) = —p. Then the above inequality becomes 
A(t)" (ve — Fe) = 0 (5.13) 


Simply put, we choose the final costate to be a vector opposite to the normal 
p and pointing toward the convex set of final states (see Figure 5.3).* The 
first component of the costate, Ao, is a constant, as we show next. 


Constancy of A, 


Consider the set of linear first order differential equations 
O ete (5.14) 


which is called the adjoint of Equation (5.10). Note that gy is gy evaluated 
at (y, à). Expanding this set of adjoint equations, 


i do Og: g2 — gn > aN 

o Oyo Oyo OY v OYo ° OYo 

n 0go O91 0g2  Ogs n OH 

1 E A ee a) E brs we eee m 1 CO 

— _ | Oy, Oy, yı 0y1 _ _| dy 

i 0m 0m 9m Əm) |y aH 
É OYn Oyn OYn Yn (y,à) i Yn ($, 1) 


where the last equality follows from the definition of the Hamiltonian. We 
rewrite this equality as 
OH : dA; OH 


A=-= or A= =— , 420,L..;n 5.15 
OY lis dt Oyi | (5,4) pos 


Since H does not depend explicitly on yo, the first adjoint equation is Ào = 0, 
thereby implying that Ay is a constant. 


Specification of Ay 


If Ao is zero, then the Hamiltonian would be independent of the objective 
functional. Consequently, the optimal control problem would have nothing to 
do with the objective functional. Since an objective functional is essentially 


* In the original derivation of the maximum principle, Pontryagin (1986) chose the final 
costate along the normal p, i.e., away from the set of final states. 
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relevant to an engineering optimal control problem, we implicitly assume that 
Ay is not zero. It could be zero in a rare problem where the solution is 
independent of the objective functional, i. e., the latter is inconsequential. See 
Kamien and Schwartz (1991) for an example. 

In practice, therefore, we expect to get solutions with any non-zero Ag and 
skip the exercise to prove that Ao is surely a non-zero constant. Nonetheless, it 
still needs to be specified. We make the simple choice, Ao = 1, and introduce 
it in Inequality (5.13) by dividing both sides by —po to obtain 


1 pi/po ..  Pn/po (Yr— $0) 20 


¿PI (A 
AT (ts) 


Considering y; = y(í;), i.e., the final state due to a pulse perturbation in 
the optimal control, we can rewrite the above inequality as 


Al'(y-$)20, at t=& (5.16) 


5.4.5 Minimality of the Hamiltonian 


Next, we will show that the dot product A! (y — Y) in Inequality (5.16) is 
constant so that the inequality also holds at ti, the time of pulse perturba- 
tion. This outcome will reveal that at any time when the optimal control is 
continuous, the corresponding Hamiltonian is minimum, and any control per- 
turbation does not decrease the Hamiltonian further. Finally, we will extend 
this minimality to times when the optimal control is not continuous. 


Constancy of the dot product A' (y — y) in [tu te] 


Let Ay = (y — y). Then utilizing Equations (5.10) and (5.14), the time 
derivative of the dot product A! (y — Y) is 


d A A 
—(A'Ay) = ——Ay c A! — = (-g} A)! Ay +A' (gg Ay) 
=-A gyAy +A gyAy = 0 


Hence, the dot product A! (y — Y) is constant so that Inequality (5.16) holds 
throughout the time interval [t;, tf]. In particular, 


A (ti) [y(t1) — $t) > 0 
which yields 
A” (t1) get), v] — ely(t1), ât) >0 


after substituting for [y (t1) —y (t1)] from [Equation (5.8), p. 133] and canceling 
out the positive time duration Atı. In terms of the Hamiltonian as expressed 
by [Equation (5.4), p. 128], the above inequality becomes 


Hly(t1), A, v] > Aly(t1), A, à(t1)] 
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where v is any admissible control value at tı, including (ti). Hence, the 
above inequality shows that at ti, the Hamiltonian achieves the minimum 
with the optimal control function. 

Note that tı is any arbitrary time instant when the optimal control is contin- 
uous. Hence the optimal control, whenever continuous, minimizes the Hamil- 
tonian. 


5.4.5.1 Minimality Even When a(t) Is Discontinuous 


Belonging to the class defined in Section 5.2.1 (p. 126), the optimal control ú 
could be discontinuous in the interval [0, ts]. Therefore, we need to show that 
even when ú is discontinuous, it minimizes the Hamiltonian. We will derive 
this result in the following three steps: 


1. We will show that the optimal Hamiltonian H(t) = A! (t)g[y (t), à(t)] 
is 


a. constant in a subinterval in which the optimal control ú(t) is con- 
tinuous, and 


b. equal to a certain function h(t) — an infimum of H with respect 
to the value of control — in the subinterval. 


2. We will show that the function A(t) is continuous throughout the interval 
[0, te] 2 


3. We will utilize the above steps to show that H(t) is 


a. defined, and the Hamiltonian H(t) is minimized by à even when ü 
is discontinuous, and 


b. the same constant throughout the time interval [0, tg]. 


Step la Given a closed subinterval [t;,t;] where ú(t) is continuous, we wish 
to show that H(t) is constant. 


Continuity of the Hamiltonian 


Observe that the Hamiltonian defined as 
H(t) =X" (Ogly(t), u(t)] 


is a continuous function of t in the subinterval [t;,t;]. The reasons are as 
follows: 


1. Both g(y, u) and gy(y, u) are continuous in the space (yi, yo. ... , Yn, U) 
as assumed in Section 5.3.1 (p. 128). 


2. As specified, ú is a continuous function of t in the subinterval [t;, t;]. 
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3. Due to the above two reasons, y and A, being respective solutions of 
Equation (5.3) on p.128 and Equation (5.14) on p.136, are continuous 
functions of t. 


As a consequence, H is a continuous function of t in the subinterval [t;, t]. 
Next, we examine a function defined as 


H(t) = HSAH A] and Htc) = H[¥(t), A(t), alte) 


for some fixed arbitrary time te in the subinterval [t;, t;]. 


Properties of H(t,t.) and its Time Derivative 

As a particular case of the continuous Hamiltonian, H(t,t.) is a continuous 

function of t in the subinterval |t;,t;]. Moreover, observe that 

BHO 9 DEW ADA IO DEI 
“a yi dt 9X; dt 

dA; 

dt 


e 
c 
~ 
> 
= 
© 
ce» 
+ 
o 
= 
a 
ES 
—— 


ERIS SE rail, teo) (547) 


is also a continuous function of t in the interval [t,,t;] for the same reasons 
given previously for the continuity of the Hamiltonian. 

At t = te, the time variable in H(t, te) is te throughout, and we can apply 
Equation (5.5) on p. 128 and Equation (5.15) on p. 136 with y = y and u = ú 
to obtain 


OB(t,t.) _ d OH (t, te) dí 
Oy; dt OA; E dt’ 


t=te t=te 


1=0,1,...,n 


The above two equations, when substituted in Equation (5.17), yield 


dH(t, te) 


dt =a 


t=te 
Since the derivative d/dt[H(t, t.)] is continuous, for each e > 0 there exists 
a 6 > 0 such that |t — te| < 6 implies 
dÉ(tt.) dH (t, te) "T 
dt dt 
t=te 
where the second term in the above inequality is zero from the last equation. 
Hence, the above inequality simplifies to 
dH(t, te) 
= AAE 5.18 
esc a E (5.18) 
Now we are ready to utilize the properties of Ê (t, te) and d/dt[H(t, te)] in 
conjunction with the minimality of the Hamiltonian. 
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Minimality of the Hamiltonian 


Being continuous in the subinterval [¢;,t;], à minimizes the Hamiltonian so 
that 

H[$(t), A(),à(t] < H(t), At), ú(to)] 

See AAA w 


H(t) H(t,te) 
Subtracting H(t.) from both the sides of the above inequality, 
H(t)— H(t.) < H(t,t.) — H(te) (5.19) 
Depending on t we have the following two cases: 
1. t> te for which we will prove H(t;) < H(t;), and 
2. t « t, for which we will prove H(t;) > A(t). 


These cases, when proved, will together imply that H (t;) = H(t;) because t 
could be on either side of te. 


Case 1 Here t > te. Since Ht, te) is a continuous function of t, the Mean 
Value Theorem for derivatives (Section 9.14.1, p. 276) yields 


" dH (t, te 
H(t, te) — H(te, te) = OEC (t — te) (5.20) 
t=tg 


for some tg in between t and te. The above equation, when combined with 
the right-hand inequality of Inequality set (5.18), yields 


H(t, te) — H(te) < elt — te) 
Comparing the above inequality with Inequality (5.19), we get 
H(t) — H(t.) < e(t — te) (5.21) 


where both t and te lie in the closed subinterval [t;, t;]. 
Now consider the auxiliary function 


F H(t;) - H(t) 


tt; 
tj—t t-t) 


which is continuous since both terms on the right-hand side of the above 
equation are continuous functions of t in the subinterval. As a consequence, 
H(t) must have a minimum at some time, since a continuous function in a 
closed interval achieves a minimum as well as a maximum according to the 
Weierstrass Theorem (Section 9.18, p. 278). Let the minimum of Ha(t) be at 
tmin So that 

Haltmin) < Ha(t) 
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in the subinterval. Expanding the above inequality, we get 


At) — B(t) || H(t) — Å (tmin) 


22 
a ~~ Mela Ped) 
Now Inequality (5.21), which is also satisfied for te = tmin, provides 
A(t) — H(tmin 
AS e (5.23) 


t— tmin 
From Inequalities (5.22) and (5.23), 


B0)- 109 <e or É(tj)— É(t;) < (tj — ti)e 


Since both (t;—t;) and e are positive real numbers that can be made arbitrarily 
small, . . . . 


Case2 In this case t < te. The Mean Value theorem for derivatives provides 
[compare with Equation (5.20)] 


dH (t, te) 


A(t.) — H(t, te) = ri 


(te — t) 
t=ta 
Combining the above equation with the left-hand inequality of Inequality 
set (5.18) and Inequality (5.19) yields 
Hi) - HD > —elte — t) (5.25) 


The auxiliary function H,(t) must also have a maximum at some time, say, 
tmax, SO that 
Habia) 2 Ha(t) 


in the subinterval. Expanding this inequality, we get 


Bt) - H(ti) - H(t) — Å (tmax) 


5.26 
tj — ti 7 t — tmax ( ) 
Now Inequality (5.25), which is also satisfied for te = tmax, provides 
A(t) — B (tua 
H(t) — H(tmax) - _ (5.27) 


b= tmax 
Altogether, Inequalities (5.26) and (5.27) yield 


Bü) = Fe) - 2) >-e or É(t)— A(t) > —(t; — tie 


Since both (t;—t;) and e are positive real numbers that can be made arbitrarily 
small, . : . n 
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Constancy of HI in [t;,t,] 


Since t is arbitrary within the closed subinterval [t;,t;], the intersection of 
Inequalities (5.24) and (5.28) yields H(t;) = H (t;). Hence if t; is an arbitrary 
time in [t;, tj], then 

A(ti) = H(tk) = H(t5) 
By necessity therefore, H is constant throughout the subinterval where the 
optimal control is continuous. 


Step 1b Consider the function h(t), which is the infimum or the greatest 
lower bound (Section 9.4, p. 269) of the Hamiltonian with respect to the value 
of the control in the set U of admissible control values. Hence, h(t) is the 
minimum value of the Hamiltonian at time t, if the minimum exists for some 
control in U. Otherwise, h(t) is the greatest value of the Hamiltonian, which 
is less than all Hamiltonian values obtainable from U. In either case, we 
specify the infimum to occur for u = w at any time t so that 

h(t) = inf H[$(),A(),u(] = Aly), AD, wE) (5.29) 
Note that w(t) is simply an m-dimensional coordinate of control that provides 
the infimum of the Hamiltonian at time t. We now show that whenever ü is 
continuous, 


Let à be continuous at time t;. Then 
H[$(ti), (t), ú(t,)] > A(t) (5.30) 
by virtue of Equation (5.29). Now consider an admissible control function 
u[(t,w(t;), which is ú(t) having a pulse perturbation w at time t; when 


ú is continuous. At such a time instant, we have already shown that the 
Hamiltonian is minimum when the control is optimal. Thus, 


HS (ti), At) At] < Hily(ts), A), ul(t;, w(t:)]) 


Since u[(t;, w(t,)] = w(t;), the right-hand side of the above inequality is h(t;) 
as per Equation (5.29). Therefore, 


Hly(t;), A(ti), W(ti)] € h(t;) (5.31) 
The intersection of the above inequality with Inequality (5.30) yields 
A(t) = H$ (t), At), ú(t)] or H(t) 


at any time t when û is continuous. 
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Step 2 We need to show that h(t) is continuous in the time interval (0, tf. 
Thus, for each e > 0 there exists a 6 > 0 such that |s — t| < 6 implies 


|h(s) — h(t)| < e 
In terms of the function denoted by 
H(t, u(t)| = H[$ (t), At), u(t)] (5.32) 


we can write 


h(t) = H[$(t), A(t), w(t)] = Alt, w(t)] (5.33) 


Because of the continuity of H (see p. 138), H is continuous in the space of 
time and control. Given a time instant s, for each e > 0 there exists a 6; > 0 
such that the norm of the variable vector [s — t, u(s) — w(t)]' less than 61 
implies 

His, u(s)] -Hie w(t)] | = [His u- 0| < e 


The above inequality for u(s) — w(t) particularly yields 
?t(s, w(t)] < h(t) + € 
From the definition 


h(s) = inf Hly(s), A(s), u(s)) = Als, w(s)) € Als, w(t)] 


Combining the last two inequalities, we obtain 
h(s) < h(t) + e (5.34) 


which establishes the upper semi-continuity (see Section 9.2.1, p. 268) of h at 
t. To complete the proof of the continuity of h(t), we need to prove the lower 
semi-continuity of h at t, i.e., 


h(s) > h(t) —e for |s—t|<62>0 (5.35) 


We prove it by contradiction. Suppose there is no 62 > 0 for which the above 
set of inequalities is true. It means that in any time interval |s — t| < 62, we 
have 

h(s) = 'H|s,w(s] x h(t)—e 
'The above inequality for an infinite sequence of progressively smaller intervals, 


[sn — t| < 1/n; n=1,2,..., can be written as 


h(sn) = Hlsn, wa] € h(t) —e 


where s,, and w,, are the respective values of s and w in the n-th interval. Be- 
longing to the set of admissible control values, the sequence of w,, is bounded. 


144 Optimal Control for Chemical Engineers 


The Bolzano-Weierstrass Theorem (Section 9.17, p. 277) assures that this se- 
quence contains a subsequence that is convergent on some value, say, wo.* As 
Wn approaches wo, Sn approaches t, thereby implying 


H(t, wo] < h(t) — € 


which contradicts h(t) being the infimum. Hence, Inequality (5.35) is true. 
Combining it with Inequality (5.34) for 6 as the lower of 01 and 62, we obtain 


|h(s) — h(t) «e for |s—t| «ó 


thereby proving the continuity of h(t) in the time interval [0, £j]. 

Finally, we prove that the minimality of H extends to even those times when 
the optimal control is discontinuous. Furthermore, H is constant throughout 
time interval [0, tp). 


Step 3a Let ú be discontinuous at a time t; somewhere in the open time 
interval (0,tf), as shown in Figure 5.4. Since h(t) is continuous throughout 


h(ti)  h(t2) h(ti-i) — h(ti) A(ti41) h(tn-1) (tn) 

h(t) FA es 
Ho Hi 3 i-i "MS Hi Hn-i: Ha 
t t ti lo d lu în a 
A 1 2 1 , +1 1 a 


Figure 5.4 The function h(t). Solid dots correspond to times when ú is not con- 
tinuous 


the interval, and equal to H(t) whenever a(t) is continuous, 


lim h(t) = h(t) = lim A(t) 


t—ti— 


Note that the number of t;s is finite, and A(t) is equal to H(t) when u(t) 
is continuous in a sufficiently small neighborhood of t;, as shown earlier in 
Step 1b. Hence, the above set of equations becomes 


Jim A(t) = hti) = lim A(t) 


which shows that H(t) is defined at t; to be equal to h(t;). Thus, ú, even 
when discontinuous, minimizes the Hamiltonian. 


* If w were a continuous function of t, then wo = w(t). However, we cannot impose 
continuity on w with respect to t, hence the need for this theorem. 
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Step 3b Now H(t) is a constant, not necessarily the same, in the left as well 
as the right part of the neighborhood corresponding to t < t; and t > t;. Let 
the respective constants be Hi; , and Hi, as shown in Figure 5.4. Then the 
above set of equations becomes 


Mia = h(t;) = Hi 
which shows that H(t) is constant over the entire neighborhood, including t;. 
This reasoning, when applied to all such t;s (i = 1,2,...,n) where 
Ü «d; ba exem dope hy < $ 


yields 


Ho = h(t4) = Ay = h(t3) = H4 =--- = Bn-1 = h(tn) = En 


as shown in the figure. As a consequence, the optimal Hamiltonian is constant 
throughout the time interval. 


5.4.6 Zero Hamiltonian at Free Final Time 


If the final time is not fixed, then from Equation (5.12) on p. 134, the final 
state is 
yr = y (tt + Ate) = y(&) + ely (ts), û(te)] Ate 
— AAA 
ê 
Hence the change in the final state from the optimal time ts is 
yr — y (tt) = SAtr 
which, when substituted into Inequality (5.13), yields 
(A'g)At > 0 
Since At; is arbitrary, it is necessary that 


A's = A(é) =0 


5.A Convexity of Final States 


Consider the following control function 


a(t), 0< cA 

Ui, tı— Atı < t <t¡ (subinterval Ji) 
u(t) = 2 a(t), t< t <t- At 

U2, tg — Ab < t «t» (subinterval I2) 

a(t), t > te 
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This function has pulse perturbations vı and va in the two disjunct subinter- 
vals I; and Iz, respectively. Elsewhere in [0,£;], the function is the same as 
the optimal control. Derived earlier in Section 5.4.1 (p. 131), the state change 
at time tı is 


Ay(t)-— gýt), v] — ely(t1), 4(t1)] Ati (5.8) 
This state change, as per Equation (5.11) on p. 133, evolves to 
Ay(t2 — At2) = P (tz — At3)W (t) Ay(t1) (5.36) 


at time (tg — Ate), the onset of the second subinterval I2. Let us simplify 
the above equation. From Equation (5.11), W(t) depends on Ay(t) — a 
continuous function of t whose derivative exists as defined by Equation (5.10). 
Thus, W(t) is a continuous function of t in the subinterval [t, ,t2— At2]. Hence, 
V (t5 — Ata) approaches W(t2) as At» tends to zero. Using the big-O notation 
(see p. 269), we have the matrix equation 


V (t9 — Ata) = W(t5) + O(Ata) 


which, when used in Equation (5.36), yields 


Ay (te = Ata) = Vr(t5) 1 (t4) Ay(ti) + O(At3)W (t4) Ay(ti) 
= P (to) vw (ti) Ay(tı) + O(At¡Atoa) 


MM RR 
second order term 


For sufficiently small At, and Ata, the second order term vanishes, and the 
state change at (tg — Ata) — the onset of I2 — simplifies to 


Ay(to — Ata) = W(t3)W (t1) Ay(t1) (5.37) 


To determine the state change at the end of I2, we repeat the steps of Sec- 
tion 5.4.1, which is concerned with a single pulse perturbation corresponding 
to the first subinterval J4. 


Step 1 Thus, in place of Equation (5.6) on p. 132, we get for the second 
subinterval I 


y (2) — $ (82) — [y(te — Ata) — (ta — At2)] = J fe(y, u) — ena 
Ay(t2—Atz) to—Ate 


Step 2 Doing the analysis that led to Equation (5.7) on p. 133, we get 


[g(y,u) - g($.à)] dt — gly(t2), va] — ely(t2), à(t2)] At + O(AG) 
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Hence, for sufficiently small Atz, the above equation when substituted in the 
result of Step 1, yields [compare with Equation (5.8), p. 133] 


Ay(t2) = y (t2) — $(t2) ^ 8ly(t2), va] — g[$ (t2), à(t2)] Ata + Ay(ta — At2) 
dy(t2) 


In the above equation, óy(t2) indicates the state change at ta if there were 
no previous perturbation. Thus, Ay(t2) is 0y(t2) if Ay(to — Ata) is zero. 
Obviously, for the first perturbation, Ay(t1) is dy(t1) so that Equation (5.37) 
can be written as 


Ay (te = Ata) = P (to) 1 (11) dy (ti) 
Combining the last two equations, we obtain 


Ay (t2) = dy (ta) + WD (to) vw (t1) dy(t1) 


Final State 


Now from tə to tf during which the control is optimal, the evolution of Ay 
is governed by Equation (5.11) on p. 133 so that the state change at the final 
time is 


Ay (ts) = y(t) — y (ts) = D(f) P * (t2) Ay (t2) 
= Vr (ic)! (t9)0y (t2) + (Eo) (1 )0y (t1) 


In general, ifthe optimal control has pulse perturbations in n disjunct intervals 
I; (i = 1,2,...,n), then the change in final state is 
Ay (Ê) = X ®(ti)dy (ti) 
i=1 
where ®(t;) = W(f;)W !(t;). Now if the final time changes by Atp, it would 
bring about an additional final state change 
y (ts + Ate) — y (te) = gly (te), últe)] Ate 


given by Equation (5.12). Thus, the final state resulting from pulse as well as 
temporal perturbations to the optimal control is given by 


= y (tr) + Bt) gl$(t).vi] — gi (ti), (6)]. Ati + ely (6), A] Ate 


The optimal control thus perturbed is an admissible control. Let us consider 
a collection of all such controls with 
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1. arbitrarily fixed but admissible pulse perturbations v1, V2,...,Vn at a 
finite number of time instants, ti, t2,...,tn;* 


2. corresponding intervals At, Atz,... At, — each of variable size, includ- 
ing zero; and 


3. intervals of temporal perturbations at +f, which also have variable sizes 
including zero. 


In terms of a; introduced in the last equation, the final state is then 


y (tr) = $(£) + aiAt; + gly Ê), à(£)] Ate (5.38) 


i=1 


Containing all combinations of the pulse and temporal perturbations, the 
collection of admissible controls includes the optimal control, which has all 
At;s and At; of zero size. Each member of this collection results in a final 
state. The final state is optimal when the control is the optimal control. 


Convexity 


We now show that the set of all final states given by Equation (5.38) is convex. 
Thus, given any two final states y(f£r) and y (fr) of this set, their combination 


z(t) = ay(ts) + (1 —a)¥(t), O<a<l 


also belongs to this set. It means that we need to show that z(t.) is generated 
by a member of the collection of admissible controls. 

Let y(t) and y(t) be generated by respective admissible controls u(t) and 
u(t). The control u(t) has perturbation intervals 


At, Ata, ..., Atn 


and a temporal perturbation Atf. On the other hand, the control u(t) has 
perturbation intervals 


AB, Ato, ..., Ala 


and a temporal perturbation Afr. Then 


y (tr) = H(t) + 3 aAt; + gly (tr), a(t)] Ate 
$(&) = 9 (tr) +) aAt + gly (tr), a(G)] At 


* If n is infinite, then the control would be discontinuous everywhere in [0, îe] and thus not 
admissible. 
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and 


z(t1) = Y (te) + oy (ir) + (1 — 0)y (te) 


yn + 5 a; [aAt; + (1 — JAE] + g[y ê), AÊ] [ate + (1— a) AG] 


— 
i=1 " " 
Ati Ats 


Thus, z(ts) is the result of a control u(t) from the collection of admissible 
controls with non-negative time intervals At;s and Atf.* Therefore, the set of 
final states is convex. 


5.B Supporting Hyperplane of a Convex Set 


A hyperplane is a generalization of a plane in a coordinate system of a number 
of dimensions. Any point y in a hyperplane satisfies a set of linear equations 


n 
S piyi =9, i=1,2,....n or ply=a 
i=1 


where a is some constant. For a specific point y on the hyperplane, we have 


which is the equation of the hyperplane relative to y. The vector p is called 
the normal to the hyperplane at y. 

Consider a set of points, S. If and only if all points of S lie on one side of 
P, then P is called a supporting hyperplane of S. For our purpose, we 
consider S to lie on the side opposite to the normal to P at a given point. For 
a convex set of two-dimensional points, Figure 5.3 (p. 135) shows a hyperplane 
P supporting a closed set S at a point y. Belonging to both P and S, the 
point lies on the boundary of S. 

We will now show that if S is a convex set, then at each of its boundary 
point we have a supporting hyperplane, or equivalently p! (y — y) < 0 for all 
y in S. In the following two steps we show that: 


* They are non-negative since they depend on non-negative At;s, Atjs, Atf, Atr, and a. 
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1. In a closed convex set S in R”, there exists a unique point y, which is 
closest to a given point x outside the set S. 


2. If y in the set S is closest to x, then for all y in S the dot product 
(x — $)'(y — 3) € 0 and vice versa. 


3. Given any boundary point y of the set S, there exists a non-zero vector p 
such that the dot product p! (y — y) < 0 for all y in the S. This result 
means that a hyperplane supports the set $ at each of its boundary 
point. 


Step 1 Consider within S a point yo and all points y; (i = 1,2,...) equidis- 
tant from or closer to x than yo. Then finding y, a point of S closest to x, is 
equivalent to finding the minimum element of the set 


[lx-—yil; $20,12,..] 


Now the above set is closed and bounded (by 0 and ||x — yo||). In this 
set, consider the continuous function f(y;) = ||x-— yil. According to the 
Weierstrass Theorem (Section 9.18, p. 278), there exists a minimum of f, say, 
f (y), corresponding to the smallest element of the set, |x — Y |]. Thus, there 
exists a point y in S that is closest to x. 

Next, we show that y is unique. Suppose there exists a point y, equidistant 
from x. Let 

d= Ix $l = Ix — yal (5.39) 


Since S is convex, it contains 0.5(y + ya). From the triangle inequality (Sec- 
tion 2.2.2, p.26), the distance 


Ix — 0.5(9 + ya)ll = 0:51 = $) + (x= ya)ll € 05x 9)1+0.5I/(:— v.) 
d d 


Strict inequality in the above relation is contradictory since it would imply 
that 0.5($ + ya) is closer to x than the minimum distance d. Therefore, 


| — $) + G- ya) = lx — $)l + ll — ya)! 
which means that the vectors (x — y) and (x — ya) are collinear.* Thus, 
(=$) =x- ya) 


where y is some scalar. Taking the norm on both sides of the above equation 
and comparing it with Equation (5.39), we get 


Ido mim 


* Let lla + b|| = |[a||--||b||. Squaring both sides, »57., (ai +bi)? = Y, (a? -02) - 2|al|||bl]. 
Upon simplifying, Said; = a! b = [[a|||b|]. Now a'b = |[al|||b|| cos where 6 is the 


angle between the two vectors. In this case where ||a + b|| = ||a|| + ||b||, we have 9 = 0°. 
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If y = —1, then x = (y + ya)/2, which belongs to the convex set S. This is 
contradictory since x lies outside S. Thus, y = —1. Hence, y = 1, thereby 
implying that y = ya. In other words, y is a unique point of S for which the 
distance ||x — Y || is minimum. 


Step 2 Next, we show that if y is closest to x, then for any point y in S, 
(x — $) (y-Y)<0 (5.40) 


Consider a point yy = Y + a(y — y), which lies in the convex set S for 
0 € a € 1. Since y is closest to x, we have ||x — || < ||x — y»|| or equivalently, 


lx-$l^ < lx- yl? = 1% — 9) — eG = (5.41) 


Using the Parallelogram Identity (Section 9.21, p. 280), 


lx — $) — aly = HI? = Ix — $1? + e?lly — $11? — 20(x — $)" (y — $) 
Using the above equation in Inequality (5.41), we get 
(x-3) (v -3)€(o/2ly — yl", 0<a<1 


In the above inequality, taking the right-hand limit of a to zero, we get the 
desired result, 
(x — 5) (y — 3) <0 
Finally, we show that if (x — $) ' (y — $) < 0, then y is closest to x for any 
point y in S. Consider 


2 x a 112 2 112 ^2 7 x 
lx — yl = le — $) = y SIE > Ix — $0. + lly = FIP 2(x — 9) ' (y — y) 
where the right-hand side follows from the Parallelogram Identity. Since we 
are given 

(x —3) (y-Y)<0 
the last term of the above equation is positive or zero. Hence, 


Ix- yl? > lx- SI? or |x- yll > lx- $1 


which implies that y is closest to x. 


Step 3 In terms of a unit vector p = (x — $)/|x — ||, Inequality (5.40) 
becomes 


where we denote the right-hand side, a dot product, as a scalar a. But 
a « p! x since 
P 


p'x-a-(x-$)'(x-$) = [Ix — $]? » 0 
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Consequently, p'y < p'x. In other words, given a point x; outside the 
convex set S, there is a unique point y; in S, corresponding to the unit vector 
pi = (xi — Y ¿)/l|xi — Y ¿l|, such that 


pi y < Dj Xi 


for each y in S. 

Now consider an infinite sequence of x; tending to a point y on the boundary 
of S. The corresponding sequence p; of unit vectors is bounded. According 
to the Bolzano-Weierstrass Theorem (Section 9.17, p. 277), this sequence has 
a subsequence converging to a limit, say, p whose norm is also unity. Hence, 
as i tends to infinity, the above inequality becomes 


pycp'$ o p'(y-5)«0 
Including the case y — y, we finally have 
p'(y-$)X0 
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Chapter 6 


Different Types of Optimal Control 
Problems 


In this chapter, we engage with different types of optimal control problems 
and derive the necessary conditions for the minimum. The constraints in 
the problems are handled using the Lagrange Multiplier Rule and the John 
Multiplier Theorem. Since derivatives are available in most of the engineering 
problems, we assume the functions involved are sufficiently differentiable and 
the constraint qualifications explained in Chapter 4 are satisfied. Simply 
put, we take for granted the existence of a set of Lagrange multipliers in 
the augmented functional incorporating all constraints of an optimal control 
problem. 


6.1 Free Final Time 


We first consider optimal control problems in which the final time is free, i. e., 
not specified or fixed. Thus, in addition to finding optimal control functions, 
we need to determine the optimal final time as well in these problems. 


6.1.1 Free Final State 


This is the optimal control problem of Section 5.2 (p.126) in which both 
the final time and the final state are unspecified or free. Using vectors, the 
objective is to minimize the functional 


te 
r= | Fiy uat (6.1) 
0 
subject to the autonomous ordinary differential equations 


y = gly(t), u(t)] (6.2) 
with the initial conditions 
y(0) = yo (6.3) 
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Based on the Lagrange Multiplier Rule (see Section 4.3.3.2, p. 103), the above 
problem is equivalent to minimizing the augmented functional 


te 
j=] Pid ae di (6.4) 
0 


subject to the initial conditions. The vectors y, y, g, and X are each of 
dimension n with the first element indexed 1. For example, 


m 
A= A Aa or An 
Variation of J 
We begin with the definition of the Hamiltonian 
H=F+A'g 
which is the same as Equation (5.4) on p.128 with Ay = 1 and go = F. In 
terms of H, the augmented functional becomes 


te 


J= [(H —Aly)dt 
/ 


The variation of J is given by 


te 
0 = 


+ H-—A'y Gt 
te 


te 
=| Hy by + HL 6A + HI du—ATOy —$78À dt+ H-ATy ót 
te 


0 — a 
0J/0tg 


where the last term above is the partial derivative of J with respect to tf 
resulting from the application of the Leibniz Integral Rule (see Section 9.10, 
p.273). Applying integration by parts to the integral of A! dy, we get 
te te te 
te e . 
¡ESTOS Alóy ZEE Alóy SE 
0 te 
0 0 0 


where in the last step we have made use of the fact that óy(0) must be zero 
since y (0) cannot vary — it is fixed at yo by the initial condition. Substituting 
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the above equation in the expression for 6J, we obtain 


te 
óJ = J (Hy +A) dy + (Hy — y) 6A - H1óu dt — XA (tj) óy(tz) 
0 


+ H-A!y D (6.5) 
Note that dy(t¢) is the variation in the optimal state y at time tr, i.e., 
dy (te) = y(te) — y (t1) (6.6) 
This variation is different from the variation in the final state 
dy: = y (tr + ôte) — Y (tg) (6.7) 


which also incorporates a change dt: in ts. Figure 6.1 illustrates both of these 
variations for a single optimal state y. To incorporate the specifications of free 
final state and free final time, we need to introduce dyf in Equation (6.5). For 
this purpose, we obtain a first-order approximation of dy¢ in terms of óy(t:) 
as follows. 

The variation óy; can be expressed as 


Oye = y (tr + dts) — Y (tr) = y (tr) + y(te) ote — Y (tr) 
= dy (tt) + y(te)dte 


where we have used the first order Taylor expansion for y(t; + dtr). Next, 
differentiating Equation (6.6) with respect to t, we obtain 


(tc) = y (tr) + oy (t0) 


0 t te te $ Str 


Figure 6.1 Relation between óyr and óy(t:) 
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After combining the last two equations and discarding the second-order term 
dy (t1)0tr, we get 


dy: = oy (t) + ¥(te)dte or Sy(te) = dys — y (te)óts (6.8) 


Finally, substituting the above equation in Equation (6.5), we obtain at the 


minimum of J [where 6J = 0, u = à, y = y, and, in particular, y (tt) = y (t£)] 
te 
/ (Hy +d)" dy + (Hy — y) 6A - H du dt — A (tj) dys + A(tedte = 0 
0 


oJ 


(6.9) 


Necessary Conditions for the Minimum 


Hence, at the minimum of J, and equivalently of J, the following equations 
must be satisfied: 


Á=-—Hy ya Hu =0 
Alt) = 0 H(te) =0 


along with the initial conditions, y(0) = yo. The control for which the above 
conditions are satisfied is called the optimal control ú. This result is valid 
when the following conditions are met with: 


1. the functions F, g, Hy, and Hy exist and are continuous with respect 
to y and u. 


2. the optimal control is 


2.a either unconstrained, i.e., a û; can take any value 


2.b or in the interior of a specified set containing all admissible control 
values 


Condition 2a is applicable to the present problem, which does not impose any 
constraints on the control. 

It might happen that the first condition is not satisfied, e. g., when the 
objective functional depends on |u| or the set of admissible controls contains 
discrete elements. To obtain necessary conditions for the minimum in that 
situation, we can rely on Pontryagin's minimum principle, which does not 
depend on the existence of Hy or its continuity (see Section 5.3.1, p. 128). 
According to the principle, the optimal control à minimizes H, i.e., 


Hs A, u) 2 Aly, A, ü) 


at each time in the interval [0, tf] where y and A are the state and costate at 
the minimum, respectively. 
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Example 6.1 

For the batch reactor problem in Example 2.10 (p.45), find the necessary 
conditions for the minimum of 

ir E 

T= - f cka àt, k = ko exp ——— 


RT 
0 


subject to the satisfaction of the state equation 
t = —akz" (6.10) 


with T(t) as the control, z(0) = xo as the initial condition, and tf as the 
unspecified final time. 

In this problem, x and T are the state (y) and control (u), respectively. 
The Hamiltonian for this problem is 


H = —ckz? — Mika” 


Thus, for J to be minimum the necessary conditions are as follows: 


; kE 
u a—1 UNE a = a M 
A = aka (c+ aA) i = aka" Rr (c+ aX) =0 
_p H, A RR 
" = 
Alte) = 0 x(0) = zo — ckx*+»Makx" =0 
s —9Ó 
H(ts) 


6.1.2 Fixed Final State 


Consider the optimal control problem of Section 6.1.1 (p.153). If the final 
state is fixed, say, at y (tf) = yr, then the variation óyr must be zero. Conse- 
quently, Equation (6.9) simplifies to 


te 
5 = | (Hy 3M y + (Ha y) 782 + Hu dt + H(t = 0 
0 


and the necessary conditions for the minimum of J, and equivalently of J, are 


À--H 


y: y = HA =g, Hu = 0, and H(t) =0 


along with the initial and final conditions 


y(0)— yo and y(tr) = yr 
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Example 6.2 

Let the final state in Example 6.1 be specified as x(t¢) = zr. In this case, the 
necessary conditions for the minimum are the same except A(t) = 0, which 
is replaced with x(t) = xr. 


6.1.3 Final State on Hypersurfaces 
In this case, the objective is to minimize the functional 


te 


T= / Fly(t), u(t)] dt 


subject to 
y = gly(t), u(t)), y(0) = yo 
as well as 
aly (tr)] = 0 
where q= q dq» +... dq T and each qi — 0 is an equation of a hypersur- 


face in the space of coordinates 


[Uis Y2, ---; Un): 


Thus, the final state is constrained to lie on these hypersurfaces. Given the 
initial conditions, the optimal control problem is to minimize the augmented 
functional J given by Equation (6.4) subject to the hypersurface constraint 
aly(tr)] = O. According to the Lagrange Multiplier Rule, this problem is 
equivalent to that of minimizing the further-augmented objective functional 


tr 


M = J+ula = ro. dt - uq (6.11) 


where 
A 
H= pa H2 ee Hn 


is the vector of additional Lagrange multipliers corresponding to the vector 
of constraint hypersurfaces. 
The variation of M is 


6M =6J+6p'qt+p' dq 
Because q depends on y(ts), which in turn can additionally vary with tr, 


dq = y(t) dy (te) + dyay (te)óte = y(t, d¥t (6.12) 
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at the minimum. In the above equation, qy(z,) is the matrix of partial deriva- 
tives of q with respect to y(t¢), and the right-hand side is the result of sub- 
stituting dy (tf) given by Equation (6.8). 

Substituting Equations (6.9) and (6.12) in the expression for 6M, we obtain 


te 
ôM = i (Hy +A)" Sy + (Ay — y) A+ Hl ôu dt + A (te) ots 
0 


+ôu' q — [A (t) — n dyap] yr (6.13) 


Since M has to be zero at the minimum, the necessary conditions for the 
minimum of M, and equivalently of J and J, are 


À--Hy, y=Ha, Ha=0, H(t)=0, q=0, and A(t) =p" Gye, 
along with the initial conditions y(0) = yo. 


Example 6.3 
Consider the consecutive reaction A — B 2—7 C carried out in a batch reac- 


tor. The mole fractions y1, y», and y3 of species A, B, and C are respectively 
governed by 


ji = —aiy2eP/7, y1(0) 21 
ya = aye? — agyse™/?, ya(0)=0 
js = aayaeP?/T — a3ygePS/T.  ya(0)=0 


where a; and E; (i = 1,2,3) are constants. It is desired to find the temperature 
policy T(t) that maximizes the product mole fraction ya at the final time fr, 
which is not fixed. Moreover, y3 must satisfy the two selectivity constraints 


Ya = biyi and ya = boy» at t=te 


where bı and bz are some constants. 
In this problem, T' is the control (u). The equivalent objective is to minimize 
the functional 


ys (ts) te te 
I = —ys(t) = -fan = -fisat = ji —aayaeP?/T + agyg e Es /T dt 
0 0 0 


for which we have used the state equation for the species C in the last step. 
The above objecive is subject to the satisfaction of all state equations and the 
selectivity constraints. The Hamiltonian is then given by 


2 Ei/T 2 _E1/T 


H= —aayaeP?/T + aayseP»/T — A1a01y1€ +A ayie — azyze”?/T 


+ Az agyoeP?/T — gays ePs/T 


160 Optimal Control for Chemical Engineers 


The necessary conditions for the minimum of I are 


Y —ayyze™/T yı 1 

Ya| = ajy2eFi/T — agzyze”?/T , Ya = |0 

UE a2yaeP*/T — asyseFs/T ya 0 

t=0 

a| — |Y3 — bin 0 

q2 ya — baya à 0 

Ad =A1 + Aa Jay1e Pi /T 

A —1 — Aa + Aa)ageP2/T 

"B a E Aa) aem | 

5 — 
T T T 

A O [2] [2] 

1 pa A qı qı ii E 0 1 
du]! = yi Oyo ðy | _ 

q2 092 092 
pa v 0 —b 1 
As te Oy y  0Oyl,, d i 
Oe ‚M 
Ay (tf) 
= —paibi — aba Hı + Ha 
Hr = (— Ay + A2)a1y2 E,ePV/T + + (-1 —A2+ As)aayo Ese P2/T 


T? 
+(1—As)asysE3e"9/7 — 0 


H(t) = (A + Aojaiy2eP/7 + + (- 1— Aa + As)aayae P2 / T 


EST <q 
te 


+(1 — Az)azyze 
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6.2 Fixed Final Time 


In this section, we take up problems in which the final time is specified or 
fixed. The control functions are the only optimization parameters. 


6.2.1 Free Final State 


Consider again the optimal control problem of Section 6.1.1 (p.153). If the 
final time is fixed, then its variation dt, must be zero in Equation (6.9) on 
p.156. Moreover, dy; = 6y(tr) from Equation (6.8). Consequently, Equa- 
tion (6.9) simplifies to 
te 
fa / (Hy + A) 8y + (Ha —y)6A+ H1óu. dt — A (t) dy(t;) =0 
0 


(6.14) 
and the necessary conditions for the minimum of J, and equivalently of J, are 


Á=-H,, y=H1=8, Ha=0, and A(t)-0 
along with the initial conditions y(0) = yo. 
Example 6.4 


Let the final time be fixed in Example 6.1 (p.157). Then the necessary con- 
ditions for the minimum are as follows: 


: kE 
a—1 d a 
A = aka“ (c+ A) t = —akz Pr 
Alte) = 0 x(0) = Lo 


x"(c+aX) =0 


6.2.2 Fixed Final State 


In this case, we have the optimal control problem of the last section, but with 
the final state fixed. Thus, Equation (6.14) becomes 
te 
óJ = I (Hy +A) Sy + (Ha — y) 6A +H] ôu dt=0 
0 
Thus, the necessary conditions for the minimum of J, and equivalently of J, 


are 
A=-—Hy, y-Hxy-g, and H,=0 
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along with the respective initial and final conditions 


y(0)— yo and y(t) = yr 


Example 6.5 

Let both final time and final state be fixed in Example 6.1 (p.157). Thus, 
for fixed tf and z(t) = tf, the necessary conditions for the minimum are as 
follows: 


. kE 
A = akz^- (c + A) t = —akz" — 


RT z^(c-- aA) =0 


z(te) = Tf x(0) = X0 


6.2.3 Final State on Hypersurfaces 


In this case, we have the optimal control problem of Section 6.1.3 (p. 158), 
but with fixed final time. This problem is equivalent to that of minimizing 
the augmented functional 
te 
u-[ F+aA'(-y+g) d+u'q=J+p'q (6.11) 
0 


whose variation from Equation (6.13) for fixed tr, i.e., ót; = 0, is 


te 

nues J (Hy 4- À)T ôy + (Hx — y) 0A + Hl õu dt+ôu"q 
0 
- [A] (te) — n aya] Sy (te) 


where qy(+) is the matrix of partial derivatives of q with respect to y(tr). 
Since M has to be zero at the minimum, the necessary conditions for the 
minimum of I, and equivalently of J and J, are 


À = —Hy, y = Ay =g, Ay = 0, q= O and A (te) = B dy (ty) =0 
along with the initial conditions y(0) = yo. 
Example 6.6 


Let the final time in Example 6.3 (p.159) be fixed at tf. In this case, all 
necessary conditions of Example 6.3 hold except H(tf) = 0. [] 
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6.3 Algebraic Constraints 


This section deals with optimal control problems constrained by algebraic 
equalities and inequalities. 


6.3.1 Algebraic Equality Constraints 


Consider the objective to minimize the functional 


subject to the state equations 
y = gly(t), u(t)], y(0) = Yo 
and the equality constraints 


fi(y, u) = 0, i=1,2,...,l or f(y,u)=0 
where it is provided that 


1. the functions f, fy, and fu exist and are continuous with respect to y 
and u. 


2. The number of equality constraints / is less than the number of controls, 
i.e, m. 


The reason for the above provision is that if | = m, then the constraints 
would uniquely determine the control, and there would not be any op- 
timal control problem remaining. 


According to the Lagrange Multiplier Rule, the above problem is equivalent 
to the minimization of the augmented functional 


te te 
u-[ F+aA'(-y+e)+py'f a= f(L—aT gat (6.15) 
0 


0 


where 
$ 


H= pu H2 ee Iu 


is the vector of time dependent Lagrange multipliers corresponding to the 
equality constraints, and L is the Lagrangian defined as 


L = H+p'f = F+MXg+p'f (6.16) 
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Considering that both final state and time are free, i. e., unspecified, the vari- 
ation of M upon simplification is given by [compare with Equation (6.9) on 
p. 156] 


te 
ôM = J (Ly  À) Oy +(La—y) 6A + Lip + Liu dt 
0 
— A (te) yt + L(tg)ótg (6.17) 


Since 6M should be zero at the minimum, the following equations are neces- 
sary at the minimum of M, and equivalently of J and I: 


À = -Ly y=L1=8 L,-f-0 
Lu=0 A(t) =0 L(tr) = 


along with y(0) = yo. 


Example 6.7 
Consider the isothermal operation of the CSTR in Example 3.5 (p.69). The 
reactant and catalyst concentrations, namely, y; and ya, are governed by the 
state equations 


Yi =u1 (Y — y1) — kyiya, y1(0) = y1,0 
Y2 = U2 — U1Y2, y2(0) Y2,0 


where the controls uj and uz are, respectively, the volumetric flow rate of 
the CSTR and the catalyst mass flow rate, both per unit reactor volume. 
Let us say we want u2 to be increasingly higher for smaller amounts of the 
catalyst-to-reactant ratio. For this purpose we enforce the following equality 
constraint at all times: 


2 a : —1 or = An a 
Yı pua=c Yi pua—=c 


where a, b, and c are some suitable parameters. Subject to the above state 
equations and equality constraint, the objective is to minimize the deviation 
of the state of the CSTR with minimum control action, i.e., to minimize the 
functional 

te 5 


1= | y (yi — y$)? + (ui — us)? de 
0 qe 


where the superscript s denotes the steady state value. 
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Based on Equation (6.16), the Lagrangian for this problem is 


2 
L= (yi — Yi) + (ui — UF)? + Ar [ur (Ye — i) — kay] 
i=l 
ya 1 
FĂ | = —1 
2 [uz — ury2] + 4 w^ pue 
——————ÁÀ — mm? 
f 


Then the necessary conditions for the minimum of / are as follows: 


yı ui (Ye = yı) z kyiy2 yi Y1,0 
Ya u2 — ul ya Y2 Y2,0 
t=0 
Lx 


M "TY |] 


—2(y2 = y3) + Aikyi + Agu, — = Aa 0 
yi te 
hy 


Lu, 2(u — uï) + Au(yr — 91) — A2ya 0 
Inb 
Bu (us — ub) + Ja + we 0 
_Ya 1 
Lu Yi =q puz—c B =0 
M—  — e — 
f 
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6.3.2 Algebraic Inequality Constraints 


Consider the optimal control problem in which the objective is to minimize 
the functional 


r= | Fiy(t),u(0)] at 
| 


subject to 
and the inequality constraints 
fi(y,u,t) < 0, 1=1,2,...,1 or f(y, ut) <0 


where each f; and its derivative with respect to y and u are continuous. 

The augmented objective functional for this problem is M, which is given by 
Equation (6.15) on p. 163. From the John Multiplier Theorem (Section 4.5.1, 
p. 113), the following equations must be satisfied at the minimum of M, and 
equivalently of I: 


À = -Ly y=Lh=g  L,=f<0 La = 
A(tr) =0 L(t) =0 p>0 p' Lu =0 


along with y(0) = yo where L is the Lagrangian given by Equation (6.16). 


Remarks 


The number of inequality constraints, l, can be greater than m, the number of 
controls. That the number of active constraints at any time does not exceed 
the m is assured by the constraint qualification. It requires that if p inequality 
constraints are active (i.e., f; = 0 for i = 1,2,...,p), then p should be the 
rank of the matrix of partial derivatives of f with respect to u.* Note that 
p is the number of linearly independent rows or columns of the matrix (see 
Section 4.3.2.1, p. 97). 


Example 6.8 
Let us consider Example 6.7 with its equality constraint replaced with the 
inequality 


Then the necessary conditions for the minimum are 


* This is one of the four constraint qualifications, any one of which must be satisfied to ob- 
tain the necessary conditions for the minimum in inequality constrained problems (Arrow 
et al., 1961). See Takayama (1985) for a thorough exposition. 
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e the ones in the previous example except L, = 0, which is replaced with 


and 
e u > 0, and the complimentary slackness condition, i. e., 


y2 1 
= — — —1 =0 
H yi a pua=c 


Example 6.9 
Let us consider Example 6.3 (p. 159) without the selectivity constraints and 
impose the constraints Tmin < T € Tmax on the temperature control. Thus, 
instead of the selectivity constraints, we have the two inequalities 
=T + Timin < 0 and T- Tmax < 0 
For this modified problem, the Lagrangian is given by 


L=H + pal T4 Triin) | pa(T Tmax) 


= —aayae P2 /T + asyse P/T — A121 y2e P /T + A aiy2eF/T — a3y3e P? /T 


+ Aa a2y3eP*/T — agy3e”3/T H La (7T + Tmin) + pa(T — Tmax) 


'The necessary conditions for the minimum are 


Yi a id Ui 1 

ya | = | aryze™/T — azyoeF2/T |, Ya = 0 

Y3 a3yaeP2/T — agyzeta/T ya 0 
t=0 

Xij —AM + A2 Jaye” /T A 0 

| = |(71— A2 + Aa)azel2/7|, [x] = [o 

iu (1— | M EMT l Az 0 


te 
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1 
Lr = B A + o aa y2 E,eP/T + (-1 — 2 + As aaa Ege P2/T 


+(1— Aa)aaya Ese2/T — pı + ua = 0 


Lu, E =T + Twin « 0 
Lys T m Tmax 0 
L(t) = (Aa + Ag)ayydeP/T + (—1 — Ao + Ag)aayoe”?/F 


+(1— Az)asyset?” + i (7T + Tmin) + 2 (T — Tmax) "m 0 


f 


T 
0 —T + Tmin 0 
Ha > and P E 
H2 0 H2 T— Tmax 0 
t=0 


Ly 


6.4 Integral Constraints 


Integral constraints could be equality or inequality constraints. We first con- 
sider integral equality constraints in an optimal control problem with free 
state and free final time. 


6.4.1 Integral Equality Constraints 


Consider the objective to minimize 


subject to 
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and the integral equality constraints 
te 
| 5o 02t- Ek ded. 
0 


where each F; and its partial derivative with respect to y and u are continuous. 
According to the Lagrange Multiplier Rule, the above problem is equivalent 
to the minimization of the augmented functional 


te l te 
M = J E+N y +8) dt FM pi [rank 
0 i=1 0 
te 


te 
/ H4pg!F-A'y dt-p'k = [ena park (6.18) 
Ne r” 
0 L 0 


where 


l.p= m H2 epu Iu E is the vector of time invariant Lagrange 
multipliers corresponding to the integral equality constraints; 


2. F and k are respective vectors of F;s and kis; and 
3. Lis the Lagrangian defined as 
L = H+p'F = F+A'gtp'F (6.19) 


Note that the uis are undetermined constants in contrast to A;s, which are 
time dependent. 
The variation of M is given by [compare with Equation (6.17) on p. 164] 


te 


6M = J (Ly +A) dy + (La — y) 9A+Llóu dt — A (te) dys 
0 


te 
iio / add (6.20) 
== A 
0 F 
Since 6M should be zero at the minimum of M, the following equations must 
be satisfied at the minimum of M, and equivalently of I: 
Ă = -Ly y=L1=8 Lu=0 
te 
Alt) =0 L(t) = 0 fra —k 
0 


along with y(0) = yo. Observe that L, is the vector F, and the last equation 
is the set of the integral constraints. 
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Remarks 


The integral constraints are equivalent to having additional state equations 


d n+t . 
- = Fi, Yn+i(0) = 0, Yn+i(te) = ki; = 1,2,...,1 
with state variables yj+;, i = 1,2,...,1. The reader can easily verify this 


equivalence by taking the integral on both sides of the above equations and 
applying the boundary conditions. Hence, an integral constraint is equivalent 
to a differential equation for an additional state variable with the initial and 
final conditions specified. Consequently, there is no limit on the number of 
integral constraints in an optimal control problem. 


Example 6.10 
Let us re-state the problem of batch distillation in Section 1.3.1 (p.5). The 
objective is to minimize the functional 


te 


r=- fuat 


0 
subject to the state equations 
21 = —u, z1(0) = 
i = ulza — y(21,22)),  22(0)= 
and the purity specification 


te 


no —y)udt — 0 


0 
The Lagrangian for this problem is given by 
L = —u — Aud Asu(z — y) + u(y* — y)u 


Note that while A, and As are time dependent, y is a constant. The necessary 
conditions for the minimum are as follows: 
21 —u 21 21,0 


, 
22 u(zi — y) Za] 22,0 
t=0 
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: Oy Oy 

RE | A I. 
A _ 2U Bz + pu Du i _ 0 
Oy Oy 
A == D. A 0 

2 gun + pu 825 2 " 
ee 
=z 


Lu 


—1 — A1 + A2(21 — y) + ul(y* —y) =0 


te te 


| Luat = for —yuat=o 


0 0 


L(t) = —u= Au + Azua — y) + uy" -y)u =0 


6.4.2 Integral Inequality Constraints 


Consider the optimal control problem in the last section, but with integral 
equality constraints replaced with the inequality constraints 


te 


J 5er minat s ns ped co 
0 


In this case, Equation (6.18) provides the augmented functional M whose 
variation is given by Equation (6.20). From the John Multiplier Theorem 
(Section 4.5.1, p. 113), the necessary conditions for the minimum are 


A = —Ly y=L1=8 L,=0 
Alt) = 0 L(t) = 0 >0 
te tf 
7H | ua =0 [teats 
0 0 


The integral constraints are equivalent to the differential equations governing 
the additional state variables as follows: 


dYn+i 


di =Fi, yau(0)—0, Ynyilte) < ki; 1=1,2,...,1 
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Example 6.11 
Let us consider Example 6.10 with its equality constraint replaced with 


te 


[ov puso 


0 


In other words, we want distillate purity greater than or equal to y*. Then 
the necessary conditions for the minimum are 


te 
e those of Example 6.10 except f L, dt = 0, which is replaced with 
0 


e as well as 


6.5 Interior Point Constraints 


Consider the objective to minimize the functional 


subject to the state equations 
y = gly(t), ut); y0) = yo 
and the interior point constraints 


aly(t1),t1] =0 


where 


q= q qo TT qi 
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is the vector of functions depending on y(t1) and ti, which is an unspecified 
or free time in the interval [0, tg]. With the help of Lagrange multipliers, the 
above problem is equivalent to that of minimizing the augmented functional 


tg 
m=f F+A'(-ġ +g) di--u'q 
0 


ti te 
= fuga fa aria 
0 


ti 


where H = F + A! g and 


z 
H= pa H2 ee Iu 


is the vector of time invariant Lagrange multipliers corresponding to the in- 
terior point constraints. With the help of 


a = Hy dy + Hy 6A+ H, du 


the variation of M is given by 


ti— 
om = | a-d yy OA. d+ EA y ôu- H—A'y ôt 
ti- t 
: i. 1+ 
te 


efa Al'óy —y!6A dt+ H-Aly ôte 
te 


ti 


+ 94+p4' qya, y (t1) + dy.) Y (5)0t + ae, Str 
Ne ear 


= gy )9y1 [compare with 
Equation (6.12) on p. 158] 
where 
tı- Sti —|6tu|, and tiy Sti +|6tu|, with ôtı 0 
and dy, is the variation in y for unspecified tı, similar to yf given by Equa- 


tion (6.7) on p. 155. 
Applying integration by parts to the time integrals of A! ôy and combining 
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the integrals having the same integrand, we get 
tg 


A ti- te 
GM = | (a+ Xy - 978A) at — Moy ^ — Aly + H-Xy oth 
0 
0 


ti tie 


=H=XAS. d+ H-A ýy dh 
tr 


tio 
tq'óuc-u Ay(1,)9Y1 + qi Ót1 


Expanding a and using Equation (6.8) on p. 156 as well as its equivalents for 
t1. and t¡,, which are, respectively, 


dy(ti_) = dye, — $(h-)65 and dy (ti) = dye, — (tı+)ôtı 


we finally obtain at the minimum of M 


te 
ôM = J (Ey or (56 + Hou dt 
0 
T illi 
+q ut p dy) + Alti4) Ah) yn, 


zu i p! di. + H (t4) E H (t4) Sti = A (ts) Oye + H (ts) ote = 0 


Thus, the necessary conditions for the minimum of M, and equivalently of J, 
are 


A=-—H, y = HA Hy =0 
q=0 A(t1-) = Alti) + 9/0) 4 — H(i-) = Hlt14) — wan 
A(t) =0 H(te) =0 y(0) = yo 


Example 6.12 

Based on the chemotherapy example in Section 1.3.8 (p. 15), we pose a prob- 
lem in which the ratios of immune-to-cancer and healthy-to-cancer cells must 
be equal to some desired values during the treatment period. Hence, it is 
desired to minimize the number of cancer cells (y4) at the final time tf as well 
as the use of the drug, i.e., the functional 


te te 
T= walt) + [ udt= fia ae 
0 0 


where u is the drug injection rate. The drug concentration yı and the number 
of immune, healthy, and cancer cells (yo, y3, and y4) are governed by the state 
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equations 


dy: 
dt 
dyz 
dt 
dys 
dt 
dya 
dt 


= u — Yy 


Y2ya 
Bo + ya 


—yir2 


= Vin + T2 YayoYa — yaya — oa 1— € 
= rays(1 — B3y3) — Ysysy4 — aays 1— e "1^ 


= riya(1 — Biya) — ysya — Y2yaya — arya 1— e ^ 


along with the initial conditions 
yi(0) = yio, 1—1,2,3,4 
and the interior equality constraints 


92 —a and us =b 


VA |z Ya |z 
1 1 


where tı is an unspecified time in the interval [0, tf] and a and b are certain 
specified constants. 


With the help of the state equation for y4, the objective functional becomes 


tr 


I = J riya(1 — piya) — V194Y3 — Y2YaYya — arya 1— e "^ +u dt 
0 


which needs to be minimized subject to all state equations as well as the two 
interior equality constraints. 


The Hamiltonian for this problem is given by 


H = (124) riya(l — biya) — V1Y3Y4 — V2Y2Y4 — arya 1= CUA +u 
Jc pu == yeyi] ae Wace + r222- oe uta = 
B2 + y4 


azyz l— e 9% + As rays(1— B3y3) — Ysysy4 — sys 1— e "^e 
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The necessary conditions for the minimum are as follows: 


Yi U— Y6Y1 
$ B 4 i 
Yo do in +ra 24 — eya ya — "aga — azy2 1 et 
E Ba + Ya 

Ya raya(l — Pays) — Ysysya — Asy3 1— e Vis 
Ya riya(l — Biya) — aya — V2Y2Y4 — arya 1— e 9^ 

e 

Hx 
yi Ya Y3 Ya -— =  Uio Y2,0 Y3,0 Y4,0 
A (1+ A4)oa ase” 9^ + Arye + AZaay2e 9^? + Ajagyge VLAS 
À T2Y4 = 
Aa (1 + A4)y2ya — Az ety 7 Ve 02 1 — e 9» 
Às| | (L4-24)mya — As ra(l — 20393) — ysya — aa 1— e” 
As — (1-24) ri(1 — 20134) — mys — yaya — 01 1 e ^ 
T2Y2 T2U2UA 


= cu EHE. O 
? Batya (B2+ya)? Mus SA 


——— 
= Hy 
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qı| |y2—aya} [0 
q2 ya — bya 0 
tı 
A A i 
i ! da ôn da On} | 
|o ¿| Oy Oy öy 
Aa As dq 092 092 092 " 
P Ad Oy — Oy  Oys Oa : 
ti ty >> 
Ax(t1) 
A 
- 
Aa 0 1 0 —a m 
= + 
A3 0 0 1 —b H2 
As 


tit 


6.6 Discontinuous Controls 


177 


Consider the problem described in Section 6.5 (p.172), but without the in- 
terior point constraints. In this modified problem, if the controls happen to 
be discontinuous (i.e., have finite jump discontinuities) at time ti, then the 


states defined by 


y = gly, u) 


would have different left and right-hand side derivatives with respect to time 
at tı. In other words, there would be corners at tı in the state versus time 
trajectories. See, for example, Figure 3.4 (p. 77). 

Following the approach of Section 6.5, the necessary conditions for the 
minimum in the present problem can be easily shown to be 


A= —Hy y = Ay 
q=0 A(ti_) = A(ti+) 
At) =0 H(ts) =0 


Ay, =0 
H(t1-) = H(ti+) 
y(0) = yo 


The necessary conditions at the time of discontinuity tı are 


A(t1-) = A(ti+) 


and 


H(t;i-) = H(ti+) 
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which are also known as Weierstrass-Erdmann corner conditions. They 
imply the continuity of H and A on the corners at the minimum. 


6.7 Multiple Integral Problems 


Optimal control problems involving multiple integrals are constrained by par- 
tial differential equations. A general theory similar to the Pontryagin's mini- 
mum principle is not available to handle these problems. To find the necessary 
conditions for the minimum in these problems, we assume that the variations 
of the involved integrals are weakly continuous* and find the equations that 
eliminate the variation of the augmented objective functional. 


Example 


To illustrate the above approach, let us consider the optimal control problem 
of determining the concentration-dependent diffusivity in a non-volatile liquid, 
as described in Section 1.3.4 (p. 9). 

The objective is to find the diffusivity function (gas diffusivity versus its 
concentration in liquid) that minimizes the error between the experimental 
and the calculated mass of gas absorbed in liquid, i. e., 


te 2 


te L 
r= fim. - mat = | | am. dt 
0 0 0 
> ë 


(me—me) 
The calculated mass of gas in liquid is governed by the state equation, i.e., 


Equation (1.16) on p. 9, which is expressed as 


G = ADAL e+ TA E -9 (6.21) 
A O a n 
along with the initial conditions 


c(0, 0) = csat (t = 0) (at the gas-liquid interface) 
c(z,0) =0 forO<z<L 


* Recall from Section 4.3 (p. 88) that it means having continuous partial derivatives of the 
integrands — a precondition for the Lagrange Multiplier Rule. 
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and the boundary conditions 


c(0, t) = Csar(t 
E E 
c;(L) =0 

The state equation, G = 0, constitutes a partial differential equation con- 
straint. Applying the Lagrange Multiplier Rule, the equivalent problem is 
to find the control function D(c) that minimizes the augmented objective 
functional 

te L 
J=1+ | [rGacar 


0 0 
subject to the initial and boundary conditions. Since the partial differential 
equation is a series of constraints at each point in time t and depth z, the 
Lagrange multiplier A is a function of t as well as z. 
Applying the necessary condition for the minimum, 6J = 0, we obtain 


te L 
ar [ [6x6 96) dzat — o (6.22) 


óJ 


The above equation will result in a number of equations — all being necessary 
conditions for the minimum. To satisfy 6.J = 0, we will expand the variational 
terms and apply the same reasoning as was done in Section 3.2 (p. 58). Essen- 
tially, we will find the equations which when satisfied eliminate each additive 
term on the left-hand side of Equation (6.22). 

The first term of the integrand in Equation (6.22) is eliminated with the 
satisfaction of G = 0, i.e., Equation (6.21). 

Next, we have 


te L te L 
ól = I 2(m — m) | scade dt = | [ 34e mejõcdzat 
0 0 0 0 
and 
6G = —dé+4+ gec + gc, Ócz + Je, ÔCzz + gpóD 


where g is the right-hand side of the state equation, Equation (1.16) on p. 9, 
or the term as indicated in Equation (6.21). Note that g depends on c, cz, 
Czz, and D. 


180 Optimal Control for Chemical Engineers 


Substituting the above expressions for 07 and 6G in Equation (6.22), we 
obtain 


óJ = 2 A(me — me) + Age ÓCHA — 66 + 92.00 + 9c..0Cz2 


+gpóD \ dzdt =0 (6.23) 


Next, we simplify the terms containing 6¢, óc;, and 6czz. Integrating these 
terms by parts, we get 


te L L tg L " te 
nm = - | [seas zZ -f Noe EE 
0 
0 0 0 0 0 0 
te L ig L 
L 8 
ES dzdt — J Age. óc MESE 
0 Oz 
0.0 0 0 


te 


E " 5 : 
NES dzdt — | Age..5¢2 — 5 (Age. 0c) o 
0 0 


L 
+ | Pai (Agc..) caz bau 
0 


Note that we have applied integration by parts twice to obtain the last equa- 
tion. The last three equations when substituted into Equation (6.23) yield 


te L 


: a 92 
óJ = "m A+ 2A(me E Me) + Age = 5; Ades) + gA Gees) dcdz dt — 
0 0 a 
ôJı 
L te 
/ Mc a / A Sg da 
[s " z Ye, de Jezz = c(0, 


0 0 
—— o o — ———Ó 


6J3 6Ja 
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tf 
L 
| Age, — S TE, óc(L, t) at+ | Age,,Óc; | dt + 
Oz z 0 
0 


=L 
I E A M 
óJa4 ôJs 
te L 
E = 0 
0 0 
SÁ 
SJ 


We will now find the equations that eliminate the terms óJ, to Jg, as 
indicated in the above equation, so that the necessary condition 6J = 0 is 
satisfied for the minimum. 


Elimination of ôJı 


This term is eliminated by defining 


À = —2A( A si (A Ea ) 
= Mec — Me Je Ba 9c; 972 Jezz 


After substituting the expressions for the derivatives ge, ge,, and ge,, in the 
above equation, we obtain the costate equation 


À = —2A(me — me) ^ di. i). ADeczs + AD De (6.24) 
p p 


Elimination of 5J» 


Because the initial concentration of the gas in the liquid is known at the 
interface and is zero elsewhere, óc is zero for all z at t = 0. However, since 
the final gas concentration is not specified, we enforce 


Alz, t) 20, 0<z<L (6.25) 
to eliminate 6.J2. The above equation is the final condition for Equation (6.24). 


Elimination of 6J3 


Now c(0,t) is specified to be the equilibrium concentration Csat(t) at the in- 
terface throughout the interval [0, tf]. Thus, óc(0, t) is always zero so that 6J3 
is zero. 
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Elimination of ôJ4 


Since c(L,t) is not specified, óc(L,t) is not necessarily zero. Therefore, we 
eliminate 6J4 by specifying 


A(L, 20, 0<t<tg (6.26) 


which is the first boundary condition for Equation (6.24). 


Elimination of 6Js 
The above equation along with the enforcement 
(0,0) 20, O<t< ti (6.27) 
eliminates àJ5. This specification is the second boundary condition for Equa- 
tion (6.24). 
Elimination of 6J¢ 
Elimination of this term requires that the coefficient of 6D in the integrand 
be zero, i.e., 
c 1, 
Ago =A 1+ — c+- =0, 0<S2<L, 0<t<t (6.28) 
p p 
The coefficient Agp is the variational derivative of J with respect to D. 
To summarize, the necessary conditions for the minimum are 
1. The state equation, Equation (6.21), and its initial and boundary con- 
ditions 
2. The costate equation, Equation (6.24), and its final and boundary con- 
ditions 


3. Equation (6.28), which is the stationarity condition. 
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Exercises 


6.1 Utilize the results of Section 6.1.1 to find the necessary conditions for 
the minimum of 


te 
T= Fito] + | Pye Ola 
0 
subject to 
6.2 Show that for L defined in Equation (6.16) on p. 163, 


dL OL 


de Ot 


at the minimum. 


6.3 For the batch reactor of Example 6.3 (p. 159), find the necessary condi- 
tions to maximize the final mole fraction of the intermediate product, ya(tf), 
with the specification y3(ts) = ya. 


6.4 Find the necessary condition for the minimum in Example 6.7 (p. 164) 
with additional constraints 


Ui > Uimin and ug € U2 max 
throughout the time interval [0, tr]. 
6.5 Repeat Exercise 6.3 in presence of the following selectivity constraints: 
y2 = bıyı and Ya = boys 
instead of y3(ts) = ya. 


6.6 Find the necessary conditions for the minimum in the heat exchanger 
problem described in Section 1.3.3 (p. 8). 


6.7 Find the necessary conditions for maximum oil production in the Vapex 
problem described in Section 1.3.7 (p. 13). 
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Chapter 7 


Numerical Solution of Optimal 
Control Problems 


The solution of an optimal control problem requires the satisfaction of dif- 
ferential equations subject to initial as well as final conditions. Except when 
the equations are linear and the objective functional is simple enough, an an- 
alytical solution is impossible. This is the reality of most of the problems for 
which optimal controls can only be determined using numerical methods. 

In this chapter, we introduce the gradient and penalty function methods, 
which are quite effective in solving a wide range of optimal control problems. 
Given initial guesses, these methods help in determining the local optimum 
(i.e., minimum or maximum) of the objective functional of a problem. We 
cannot discount the possibility of having a number of local optima in an 
optimal control problem. To strengthen the globality of the optimum, we 
need to apply these methods with several initial guesses and compare the 
resulting optima. Note that finding the maximum in a problem is equivalent 
to finding the minimum of the negative of the objective functional. 


7.1 Gradient Method 


We introduce this method to determine the minimum in the optimal control 
problem of Section 6.1.1 (p.153) in which both final time and final state are 
free. 


7.1.1 Free Final Time and Free Final State 


The objective of the problem is to find the vector of control functions u(t) 
and the final time t; that minimize 


= 
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subject to 
y = gly(t), ut), y(0) = yo 
The above problem is equivalent to minimizing the augmented functional 


te 
j=] FA Grio dt (6.4) 
0 
subject to y(0) = yo. From Section 6.1.1, the necessary conditions for the 


minimum of J are 


y=8, y(0)=yo, A=—Hy, A(t) 20, Hy=0, and H(t)-0 
Ne —___” ÎN 


— 
state equations and costate equations and 
initial conditions final conditions 


In the gradient method, the state and costate equations are solved using 
initial guesses for the control u and the final time tf. The guessed u and tf 
are then improved using, respectively, 


1. Hu, the variational derivative of J with respect to u at each time in the 
interval [0, te], and 


2. H(t,), the partial derivative of J with respect to tr. 


The above partial derivatives are the components of the gradient of J. The 
state and costate equations are solved again using the improved u and tr, 
which are further improved using the resulting gradient of J. This procedure, 
when repeated, leads to the satisfaction of the remaining necessary conditions, 
Hu = 0 and H(ts) = 0, and thus to the minimum. The step-wise iterative 
procedure of the gradient method is as follows. 


7.1.2 Iterative Procedure 


Guess control functions and the final time in the beginning, and carry out the 
following steps: 


1. Integrate the state equations forward to the final time using the initial 
conditions at t = 0 and the control functions. 


2. Calculate the objective functional using the control functions, the final 
time, and the corresponding state as determined by the state equations. 


3. Integrate the costate equations backward to t = 0 using the final condi- 
tions at the final time, the control functions, and the state determined 
in the previous step. 


4. Improve the control functions and the final time using the gradient infor- 
mation and repeat Step 1 onward until there is no further reduction in 
the objective functional. 


The last step employs the improvement strategy that is described next. 
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7.1.3 Improvement Strategy 


With the state and costate equations satisfied at the end of Step 3 above, the 
variation of the augmented objective functional becomes [see Equation (6.9), 
p. 156] 
te 
ôJ = Eu dt + H (t;)ót; (7.1) 
0 


In the above equation, Hu or the coefficient of du is the variational derivative 
of J with respect to u at a given time in the interval [0, tf]. Similarly, H (tẹ) 
or the coefficient of 6t is the partial derivative of J with respect to the final 
time tf. The improvement strategy is to achieve maximum reduction in J by 
changing u(t) and tf, which are not necessarily optimal. The straightforward 
way is to change u(t) and tf, respectively, by 


óu(f) = —eHa(t) and dtp = —eH (ts) (7.2) 


where e is some positive real number. With these changes, the variation in J 
becomes 


te 
óJ = —e | inso dt + H? (tẹ) 
0 


For sufficiently small e, the changes du, and dts are small enough so that the 
change in J is given by 


J(u + Ou, te + dts) — J(u, te) = 6J 
E AS: e 
Jnext 


The above change in J is the most negative for some optimal e. Hence, based 
on this e, the most improved (i. e., reduced) functional value Jnext results from 


Unext (1) = u(t) + du(t) and te next = te + ote (7.3) 


and the corresponding state Ynext. Thus, Equations (7.2) and (7.3) form the 
strategy to improve u and tf. Since u and tf (as well as unext and tf next) along 
with the corresponding states satisfy the state equations, 


J=I and Jnext = [next 
from the definition of J in Equation (6.4) on p. 154. 


7.1.3.1 Numerical Implementation 


To implement the improvement strategy numerically, we split the time inter- 
val into N subintervals of equal length and use numerical integration [e. g., 
composite Simpson’s 1/3 Rule given in Section 9.12 (p. 275) for even number 
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of subintervals] to calculate 6J as well as J. Thus, J or I is rendered into a 
function dependent on the vector of optimization parameters 


p= ulto) uti) ... uw) tr. (7.4) 
where to,t1,...,ty form the time-grid of (N + 1) equispaced grid points in 
the time interval [0, tr]. 

However, a complication arises with this approach. An improvement in t, 
changes the time grid, thereby requiring the estimation of controls and states 
on the new time grid for the next round of improvements. We avoid this 
situation by linearly transforming the independent variable t in the variable 
interval [0, tf] to a new independent variable o in the fixed interval [0, 1]. 


Transformation of the Independent Variable 
Let the new independent variable o be given by the linear relation 
o=at+b 


where a and b are some unknown constants. Substituting, respectively, the 
initial and final values of o and t in the above equation, we obtain 


1 
O=b and l=atp+b => aa 
f 
Hence 
oi do d and dy; dy;de _ dy; 1 
(od dé tg dí de dt dots 


Based on the above relations, the objective of the optimal control problem is 
to find the control u and the final time tf that minimize 


te 1 
do 
r= [rv at = | rivis). uto)]de 
0 0 
subject to 
qo > tely(o),u(o)), y(0) = yo 
O 
The Hamiltonian is then given by 


H=t(F+A'g) 


Necessary Conditions for the Minimum 

The above problem is equivalent to minimizing the augmented functional 
1 1 

ted tF +A! -F Lig d= f a-a do 
0 


d o 
0 
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subject to y(0) = yo. The variation 6J is given by [compare with Equa- 
tion (6.14) on p. 161] 


1 
a= / (Hy + Ayr by + (Ha — SETE +Hlóu do — A (1) óy(1) 
on on 
0 


si I E dest (7.5) 


where J;,, as indicated above, is the partial derivative of J with respect to 
the final time tf. Since 0J = 0 at the minimum, the necessary conditions for 
the minimum are 


dy dA 
— = tg, y(0) — yo 77 — Hy, A(1) =0 
do do 
O — ee” 
state equations and costate equations and 
initial conditions final conditions 
1 
H 
Hy = 0, J, = | —do =0 
te 
0 


Improvements in u and t to Reduce J or I 


These improvements are done in Step 4 of Section 7.1.2 (p.186) where the 
state and costate equations are satisfied so that 


1 1 1 
H 
1=1= [rac and i= | Hpéudo+ | ae 
f 
0 0 


— o_o, 


Jt, 


First, we compute the integrals in the above equation numerically over the 
fixed o-interval [0,1]. The interval is split into N subintervals of equal length 
using (N + 1) equi-spaced grid points 


09 = 0, 01, 02, ine ON =l 


For example, using composite Simpson's 1/3 Rule (Section 9.12, p.275) for 
an even N, 
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N N 
1 " 
5] = qw Hu(00)5u(00) +4 Y Ay (ci)óu(e;) + 2 Hu (o;)óu(c;)- 
i=1,3,5,... i=2,4,6,... 
Hi (oy)óuloy) + 
1 N N 
aye Elo) A Y H(0;) +2 H(0;) + H(ou) dt 
tr i=1,3,5,... i=2,4,6,... 
Jtg 
where the vector 
a 
VJ = Ha(00) Hu(01) m Halon) Jt 


is the gradient of J. 
Next, we consider the vector of optimization parameters, p, defined in Equa- 
tion (7.4). Let a change in p to reduce J be given by 


dp= óu(co)  óu(o1i)  ...  6u(on)  ót; dj 


According to Appendix 7.A (p. 229), the reduction in J per unit size of óp is 
maximum along the direction of — VJ, i.e., opposite to that of the gradient 
VJ. This direction is known as the direction of the steepest descent. It 
can be easily seen that Equations (7.2) and (7.3) already utilize the steepest 
descent direction to improve u as a continuous function of t, and tf. 
Now a change of magnitude ey in p along the steepest descent direction is 
given by 
dp = eo(-VJ/|VJ|) = -eVJ 
M— ë r” 

unit vector 

along -V J 
where e = €o/||VJ||. For some optimal e, the improved functional value Jnext 
obtained from 


Pnext = P + dp 


and the corresponding state Ynext is expected to be less than J as much as 
possible. Note that 
J=TI and Jnext = inext 


because the controls and the corresponding states satisfy the state equations. 
The improvement in Pnext translates to the improvements 


Unext (7%) = u(ci) — Hulo;); 1=0,1,...,N; and 


tf next = te — €Jt, 
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which reduce J. 

Subsequent reduction of J is achieved as follows. Based on the improved 
controls and final time (namely, Unext and tf next), Steps 1-3 of Section 7.1.2 
(p. 186) are repeated, and VJ is recalculated and utilized to repeat the im- 
provements and reduce J. This iterative procedure continued until the reduc- 
tion in J becomes insignificant or the norm of VJ becomes negligible. This 
minimization procedure is known as the gradient algorithm. It affords a 
simple and effective way to solve a wide range of optimal control problems. 


Remark on Gradient Improvement 


A number of methods have been developed to improve the steepest descent 
so that the minimum in a problem is attained in a least possible number 
of iterations. Since the gradient of a function is zero at the minimum, one 
approach is to use the quadratically convergent Newton-Raphson method 
(see Section 9.11, p.273) to zero out the gradient. However, the method 
needs the Jacobian of VJ, which is also known as the Hessian of the J. The 
Broyden-Fletcher-Goldfarb-Shanno (BFGS) method (see Press et al., 
2007) ingeniously utilizes the gradient information (i.e., 6p) to construct the 
Hessian and provides the optimal improvement Pnext to bring the maximum 
reduction in J. The BFGS method is known to be one of the most efficient 
and stable optimization methods. The numerical examples in this book are 
solved using this method. 


Number of Grid Points 


Note that the above numerical implementation provides a solution of an op- 
timal control problem as discrete values of optimal controls and states on the 
grid of the independent variable. The solution becomes more accurate as we 
increase the number of grid points, i.e., N. In the limit of N tending to in- 
finity, the solution has optimal controls and states as continuous functions of 
the independent variable. 

In practice, we solve an optimal control problem with increasing values of 
N up to a limit beyond which either there is no significant difference in the 
solution or the computations become very intensive. Note that larger the N 
the larger is number of discrete control values to be optimized and the harder 
it is to solve the problem. 


7.1.4 Algorithm for the Gradient Method 


Based on the above development, the algorithm is as follows: 


1. Transform the optimal control problem from the variable time interval 
[0, te] to the fixed o-interval, [0, 1] 


2. Assume a value for t; and N. Discretize the o-interval [0, 1] using N 
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equi-spaced grid points. Assume a value of the control function u at 
each grid point. 


3. Integrate state equations forward using the initial conditions and the 
control function values. Save the values of state variables at each grid 
point. 


4. Evaluate the objective functional using the values of control functions 
and state variables. Save the objective functional value. 


5. Integrate costate equations backward using the final conditions, the con- 
trol function values, and the saved values of the state variables. Save 
the values of costate variables at each grid point. 


6. Improve u and t using the gradient of the objective functional as follows. 
Change u by —eH, at each discrete point and t; by —eJ;, where e is a 
positive real number, which causes maximum reduction in the objective 
functional. 


7. Repeat computations Step 3 onward until there is no further reduction in 
the objective functional or the norm of the gradient becomes negligible. 
When either event happens, the u, tf, and corresponding state variables 
are optimal. 


The above algorithm is explained in more detail with the help of the fol- 
lowing example: 


Example 7.1 

Consider the reaction A + B —> C carried out in a fed-batch reactor. The 
volume y; of the reaction mixture, and the concentrations ya, ys, and ya, 
respectively, of A, B, and C are governed by 


dyi 

— = 0 = 

dt U1; yi (0) Y1,0 
d u = 

cs ua(yo = 82) _ koe“/“2y2y3, ya(0) = yao 
dt Yi 

dyz U1Y3 

mL ER — koe? / "2 0)— 

dt "m 0€ Y2Y3, yal ) Y3,0 
d u 

n d A y kye*/“2y2yz, y4(0) = Yao 
dt yı 


where the control 


e u is the time dependent volumetric flow rate (cm? /min) of the reactor 
feed with yo (g/cm?) concentration of A, and 
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e uz (K) is the reactor temperature as a function of time. 


It is desired to find the optimal controls uy (t) and uz(t) that maximize ya (tf), 
i.e., the final concentration of C. Thus, the objective is to minimize 


ya(ts) te 


d 
L = —ya(ts) = -fav = 7 spat 
0 0 
te 
i / DU koe" yy, dt 
Yi 


subject to the state equations. 


Transformation to Fixed o-Interval 


The equivalent problem in the fixed o-interval [0, 1] is to minimize 


1 
I= [e = — Kyaya do 
0 


yi 
—————À m 
F 
where we have introduced 
k = kper/u2 
The state equations are 

d 
LE uj, y1(0) = vio 
do 
d u — 
P =t wie — Ya) _ KyoUs , y2(0) = y2,0 

o yı 
d U1Y3 
P Exp I. KY2Y3 ; ya(0) = y3,o 

Oo yı 
dy4 U1Y4 
ae =t — + KyoU3 ; ya(0) = Yao 

Oo Yi 

The Hamiltonian is then given by 
u u — 
H = t 14 — Ky2y3 + Aur + Ag ua (yo = y2) — KY2Y3 
yı Yı 
u u 
+A3 — T Ky2y3 + As e kan \ 
Yi Yi 
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Necessary Conditions for the Minimum 


The necessary conditions for the minimum of I are 


dyi 
de ui " 
dys u1 (yo - y2) 
de ——— — 9293 Y2 
= te yı , = 
dys u1Y3 
per zx — KY2Y3 Y3 
do Ui 
d u 
aya — 194 + KY2Y3 Ya 
do = 
Hx 
dài (1—A4uiga , Azwi(yo — y2) _ Asurys 
do yi yi yi 
dA u 
— (1 — days Ag — -- ya +A3ky3 
do = te Yi 
dA u 
a = Aa) KY2 = A2KYa + Az bal + KY2 
O yı 
dA4 ui 
Sis ETE pd 
do ( A) Ui 
N 


à a As M`M=0 0 0 0? 


(1 — Aa)ya A2(Yo — Y2) — Ass 


Hi, TA) + 
= te yı yı yı 
GK 
Au, (sie reece o 
2 
1 
A 
Jt, ES. —0 
0 


The gradient algorithm to solve this problem is as follows. 


I 
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Gradient Algorithm 


1. 


Set the iteration counter k = 0. Assume iP and an even N. Obtain the 
fixed o-grid of (N + 1) equi-spaced grid points 


go(— 0), 01, 02, Aak. ON-1; on(= 1) 


We use k in a superscript to denote the iteration number. 


At each grid point, assume control function values as follows: 


i=0,1,...,N 


Integrate state equations forward from o = 0 to 1 using the initial 
conditions and the control function values uk, i = 0,1,..., N. Save the 
values of state variables at the grid points as 


Yi i 


Y2 (6: 


0; 


<Q 


¡(0;) 
i |, i=0,1,...,N 
EGO) 
t (7%) 


0; 


Evaluate the objective functional using the controls us and the state 
variables ys. Using composite Simpson's 1/3 Rule, the objective func- 
tional value is given by 


N N 
A 1 
Mele Ao +4) Ai 2M Ai + An 


i135. i=2,4,6,... 
where 
k wk k uli a kk 
A; = tF (yi, uF) = ty z^ ~koexp = YY», 1=0,1,...,N 
Ui Ug 
. Check the improvement in J for k > 0. Given a tolerable error e; > 0, 
if 


|I I1 | <a 
then go to Step 10. 


Integrate costate equations backward from o = 1 to 0 using the final 
conditions, the controls us, and the state variables ys. Save the values 
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of costate variables at the grid points as 


Af = A" (o;i) = 


6. Evaluate the gradient of J* by calculating the partial derivatives 


HÈ = HE (ci) = "IE 1—0,1,...,N 
(VE, uf, Aj ) 
1 N N 
and Ji = aL 16 +4) 742) Hi + HA 
f 1,3,5,... i=2,4,6,... 


where HE = te F(yk, uk, AF) +A) gyf, ULA); 1=0,1,...,N 


Check the magnitude of the gradient. Given a small positive real number 
€2, if the norm of the gradient 


N 2 2 2 
YO Hag(GhuhAD + JE <e 
i=0 j=1 


then go to Step 10. 
7. Improve control functions by calculating 


MES ET 


i u,i 


u i=0,1,...,N 


where e is a positive real number causing maximum reduction in IF. 


Assign 
k+1 
i 


utti + uF; ¿=0,1,...,N 


8. Improve the final time using 


and assign 


9. Increment k by one and repeat calculations Step 2 onward. 


10. Terminate the algorithm with the following result: 
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The optimal objective functional value is IF. 

The optimal control u(t) is represented by u^, i = 0,1,... 
The optimal final time is t£. 

The optimal state ¥(t) is represented by y^, i = 0,1,..., N. 


Results 


Table 7.1 lists the parameters used to solve the optimal control problem with 
the above algorithm. In this problem, the final time and final states are free. 
It is desired to find two controls, u1(t) and uz(t), and the final time tf that 
maximize the final product concentration ya (tf). 


Table 7.1 Parameters for the problem of Example 7.1 


yao 0 g/cm? yo 10 g/cm? N 100 
y3,0 40 g/cm? a —8420K 


The initial and optimal states are plotted in Figures 7.1 and 7.2, respec- 
tively. The corresponding controls are shown in Figure 7.3. 


yi/10°, Y2, Y3, Ya 


t (min) 


Figure 7.1 The initial states versus time for Example 7.1 
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With the initially guessed final time of 60 min, the initial controls pro- 


vided / = —4.12, which corresponds to the final product concentration of 
4.12 g/cm?. 


t (min) 


Figure 7.2 The optimal states versus time for Example 7.1. The final time has 
reduced 


500 
400 F 
NFL 
300 + 
3 
_ L UL. Apra ua 
3 
- 200 + "-— ûz 
3 
că pouce c 
0 L | L | Í | L | L | i 
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t (min) 


Figure 7.3 The initially guessed and optimal controls for Example 7.1 
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The application of the gradient algorithm minimized J, i. e., maximized the 
product concentration to 7.56 g/cm?. The final time was reduced from 60 to 
50.3 min. Figure 7.4 plots J versus the iteration of the gradient algorithm. 


—pooooooooo00000000000000€ 
x 
—5r x 
x 
x 
L x 
J-6 x, 
L q 
TE XXX 
x 
XX 
F XXXXXX 
-8 i | 1 | | | | i 
0 10 20 30 40 
iteration 


Figure 7.4 The objective functional versus iteration for Example 7.1 


7.1.5 Fixed Final Time and Free Final State 


The objective in this problem (see Section 6.2.1, p. 161) is to find the u that 
minimizes 


subject to 


The final time tf is specified, i. e., fixed. 


Example 7.2 

Let us consider the problem of Example 7.1 (p. 192) with the final time fixed 
at 60 min. In this case we apply the gradient algorithm on p. 195 with i fixed 
at 60 min and skip Step 8 of the algorithm. 
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Results 
For the same set of parameters and initial controls used in Example 7.1, Fig- 


ure 7.5 shows the optimal states with the fixed final time. The corresponding 
controls are shown in Figure 7.6. 


/10?, da, 93, Ya 


Y 


t (min) 


Figure 7.5 The optimal states versus time for Example 7.2. The final time is fixed 
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Figure 7.6 The initially guessed and optimal controls corresponding to Figure 7.5 
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The optimal / is —7.51, corresponding to the final product concentration 
of 7.51 g/cm?. When the final time was free and available for optimization in 
the last example, we got better results — more production in less time. The 
reason is that free final time affords more freedom (than when it is fixed) in 
the optimal control of a process. 1 


7.2 Penalty Function Method 


This is a simple method for solving an optimal control problem with inequal- 
ity constraints. As the name suggests, the method penalizes the objective 
functional in proportion to the violation of the constraints, which are not en- 
forced directly. A constrained problem is solved using successive applications 
of an optimization method with increasing penalties for constraint violations. 
This strategy gradually leads to the solution, which satisfies the inequality 
constraints. 

To illustrate the penalty function method, we consider the following optimal 
control problem. 


7.2.1 Free Final Time and Final State on Hypersurfaces 


The objective in this problem is to find the control u and the final time tf 
that minimize 


subject to 


and the /-dimensional hypersurface 


aly(tr)] = 0 


This problem is equivalent to finding the minimum of the functional 


te 
m= | F+.X'(-y+g) dt+p'q 
0 


— a 
J 
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where yp is the vector of Lagrange multipliers corresponding to the hypersur- 
face. From Section 6.1.3 (p. 158), the necessary conditions for the minimum 
of M are 


y=s y(0) = yo Ă = Hy A(t) = n dyay 
Hy, =0 H(t) =0 q=0 


The gradient algorithm as such cannot work with the above equations. With 
an assumed control if we integrate the state equations forward from the initial 
conditions, there is no guarantee that the state will satisfy q = 0 at the final 
time. 

The penalty function method addresses this difficulty by prescribing 


u = Wa 


where W is an l x l diagonal weighting matrix with all positive diagonal 
elements. With this arrangement, 


A (te) = aw dy (ts) 
aT 


and the augmented objective functional becomes 


M = J+(Waq)'q = J+q' Wq 
a 7 — = 


p penalty 
function 


where the last term, which is known as the penalty function, does not con- 
tribute to the augmented functional M as long as all constraints q = 0 are 
satisfied. 

In contrast, if a final state condition, say, q; — 0, is violated, M is enlarged 
or penalized by a positive penalty term d;;q?. For sufficiently large d;;, the 
penalty term would increase M to such an extent that its minimization would 
necessitate q; to be suitably close to zero. Because of this fact, sequential 
minimization of M each time with increased magnitude of W would lead to 
the minimum that satisfies q — 0. This minimization procedure is called the 
penalty function method, which fits easily with any optimization algorithm 
to enforce constraints. The computational algorithm combining this method 
with the gradient algorithm is as follows. 


Computational Algorithm 


To determine u and tf at the minimum, 


1. Initialize the outer counter r = 0. Choose a real number a > 1. 
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2. Set the weighting matrix W = a”1 where 1 is the | x l identity matrix 


given by 
1.0 0 
0 1 0 
1 = 
0 0 0 
0 0 1 


3. On the problem transformed to the fixed o-interval [0,1], apply the 
gradient algorithm of Section 7.1.4 (p. 191) with 
A) eat 
uT 
For r > 0, the initial guesses for the controls and the final time are, 
respectively, given by 


u'; 


2? 


1=0,1,...,N and tj 
When the gradient algorithm converges in, say, k iterations, save the 
state, the control, and the final time, i.e., 


yf, ubl ¿=0,1,...,N; and t 


à 
4. Increment the counter r by one and assign 
yË > y, u? > ul, 1=0,1,...,N; and trot 
5. Check whether the constraint q = O is satisfied. Given a positive real 
number ez close to zero, if ||q||, i. e., 
l pa? 
a Glyn) > €3 
then go to Step 3. Otherwise, the constraint is satisfied, and the values 
yi; ui, 1=0,1,...,N; and tj 


correspond to the minimum. 


Example 7.3 
Consider the batch reactor problem in Example 7.1 (p.192) with selectivity 
constraints at the final time as the vector of two hypersurfaces 


ya — by» 0 
qly(ts)] = = 
Ya — boy3 0 


Following is the computational algorithm to solve this problem using the 
penalty function method: 
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Computational Algorithm 


1. Set the counter r = 0. Initialize the weighting matrix as 


W — 


2. Apply the gradient algorithm of Section 7.1.4 (p. 191) with 


: 
ài T 
ya — bry2 a” 0 —bi 0 1 
A'(1) = lA = 
y4 — boy3 0 a” 0 —bo 1 
A3 A E p — 
o=1 qT W Oy (1) 
m 


—bi(ya m b1ya) 
=0 | —bo(ya — baya) 


2y4 — biy2 — boys 
Recall that o is the transformed time in the fixed interval [0,1], and 
o = 1 corresponds to the final time tf. When the gradient algorithm 
converges in, say, k iterations, save the state, the control, and the final 
time, i.e., 
y*, TF, i=0,1,...,N; and tf 


. Increment the counter r by one and assign 


y yl TEST; i-20,L..,N; and >t 


. Check whether the constraint q = 0 is satisfied. For a positive real 
number e3 close to zero, if 


2 2 
IP d(YN) > ea 
then go to Step 2. Otherwise, the constraint is satisfied, and the values 


Vis Le 


2) 


1=0,1,...,N; and tj 


correspond to the minimum. 
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Results 


We present the results for b; = 5 and 69 = 2 and the same set of parameters 
and initial controls as used in Example 7.1. 

Figure 7.7 shows the convergence of the penalty function method. At each 
outer iteration r, the plotted objective functional value is the optimal with cor- 
responding constraint violation quantified as ||q||. It was 17.2 at the very be- 
ginning (corresponding to the initial controls) and dropped finally to 5.9x 10-85 
at the convergence, which was attained in 11 outer iterations. 
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Figure 7.7 The intermediate, optimal objective functional Í , penalty factor a, and 
the constraint norm ||q|| versus outer iterations 


Figure 7.8 shows the optimal states obtained at the convergence of the 
penalty function method. The states correspond to the optimal controls, 
which are shown in Figure 7.9. The final time was reduced from 60 to 42.8 min. 

The selectivity constraints are satisfied at the optimal final time of 42.8 min. 
The optimal objective functional is —6.35, which corresponds to the final 
product concentration of 6.35 g/cm?. 

Compared to the problem in the absence of the selectivity constraints (Ex- 
ample 7.1), the final product concentration has reduced a bit due to the 
imposed selectivity constraints. 
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50 


Figure 7.8 The optimal states versus time for Example 7.3 
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Figure 7.9 The initially guessed and optimal controls versus time for Example 7.3 


Numerical Solution of Optimal Control Problems 207 


7.2.2 Free Final Time but Fixed Final State 


The objective in this problem is to find the control u that minimizes 
te 
I= J Fi.) dt 
0 
subject to 


y = gly(t), u(t)], y(0)=yo, and y(tr) = yr 
The final state fixed as y(tr) = yr is the vector of hypersurfaces 
aly (tr)] = y (tr) — yr = 0 
at the final time tf. From Section 6.1.3 (p. 158), the necessary conditions for 
the minimum of M are 
y-g  y(0)=yo A=-Hy Alt) =H aya) =H" 
Ha=0 H(t) =0 q=0 
considering the fact that qy(,) is an identity matrix. 
Applying the penalty function method developed in Section 7.2.1 (p. 201), 
u=Wa, A(t) =h' =q'W 
and the objective functional M becomes 
M = J*(Wq)'q = J+q Wa 
— | 


penalty 
function 


The computational algorithm to determine the optimal control is the same as 
on p. 202. 


Example 7.4 
Consider the batch reactor problem in Example 7.3 with the final state fixed 
as 
yi(te) = yip; t= 1,2,3 
instead of the final selectivity constraints. 
The computational algorithm to solve this problem is as follows. 


Computational Algorithm 


This modified problem is solvable by the algorithm of the last example using 
the vector of hypersurfaces 
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and the final costate vector 


T T T 
à Yi — Yi a” 0 0 Yi — Yi 
îi 
A (1) = |A2 = ya —Yor 0 a” 0 | =a" | yo — yat 
A3| _ ys — Var] _. | 0 0 a" juu. 
o=1 o=1 =i 
qt w 


Results 


This problem was solved with 
ye=5x10é ar =0.5  yas=0.5 (g/cm?) 


and the same set of parameters and initial controls as used in Example 7.3. 

The constraint violation in terms of ||q|| was 1182 at the very beginning 
with initial controls and converged finally to 1.1x10~° in five outer iterations. 
Upon convergence, the final values of y1, y2, and y3 were 


yir = 5 x 10? y2,— 0.4993 var = 0.4993 (g/cm?) 


and the optimal objective functional was —7.5, which corresponds to the final 
product concentration of 7.5 g/cm?. 'The optimal final time reduced from 60 
to 48.1 min. Figure 7.10 shows the optimal states obtained at the convergence. 
The corresponding optimal controls are shown in Figure 7.11. 


t (min) 


Figure 7.10 The optimal states versus time for Example 7.4 upon convergence of 
the penalty function method 
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Figure 7.11 The initially guessed and optimal controls corresponding to Fig- 
ure 7.10 


7.2.3 Algebraic Equality Constraints 


The objective in this problem is to find the control u that minimizes 


tr 


I= ESOO) dt 


subject to 


and l algebraic equality constraints 


f(y,u) =0 


It is assumed that the partial derivatives fy and f, are continuous, and the 
dimension of f is less than that of u. As shown in Section 6.3.1 (p. 163), the 
equivalent problem is to find the minimum of the augmented functional 


te te 
M = rosa ut dt = I b-i y 
0 0 
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where u are the time dependent multipliers associated with the algebraic 
equalities and 


L = F+A'g+p'f = H+p'f 


is the Lagrangian. The necessary conditions for the minimum of M, and 
equivalently of I, are 


y-g y(0) = yo 
f=0 


À = -Ly Alt) =0 
Lu=0 L(t) =0 


Il 


Since the algebraic equations f = O are not expected to be satisfied with 
an initially assumed control, we penalize the Lagrangian. The approach is 
similar to that in Section 7.2.1 (p. 201). We set u = Wf at each time in the 
interval [0, tf]. Thus, the objective functional 


te 
= Jr Í T 
m=f PA (~y +g)+ £ Wf dt 
0 penalty 


function 


is penalized through the penalty function whenever any algebraic constraint 
is not satisfied. The computational algorithm to determine the minimum is 
as follows. 


Computational Algorithm 
To determine u and tf at the minimum, 
1. Initialize the outer counter r = 0. Choose a real number a > 1. 
2. Set the l x l diagonal weighting matrix 
W=a1l 


for the (N + 1) grid points of the o-interval to be considered in the 
gradient algorithm. 1 is the l x l identity matrix. 


3. Apply the gradient algorithm of Section 7.1.4 (p. 191) with the following 
initial guesses if r > 0: 
u;; i—0,L...,N and tf 


2? 


Note In this case, the augmented objective functional is M whose gra- 
dient in the transformed o-interval [0, 1] is 


VM 2 Luloo) Lalo) ... Lulon) M, | 
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where 


L(0:) = tf F(y;,u;) T Aj g(yi, us) 


+ [f(y;, u;)] | W f(y;, us); = 0,1, .. N 
— 


— ao) 


nj 


Using composite Simpson's 1/3 Rule, given that the state equations are 
satisfied, 


N N 
1 
Mec Ao+4 A; +25) As + An 
13,5... 1=2,4,6,... 


where A; = UP (yi, u;) + [f(y;, uj] W f(yi, uj); i= 0, 1, ...9 N 
The improvements in u and tf are, respectively, —eLy at each grid point 
and —eM;, where 


1 


H 1 N N 
ho] deco Horta Y His Y H+H 
e tr i 3Nt ps m t mos " 


and A; = tf F(y;, ui, Ai) + Aj g(yi, ui, As) ; 1=0,1,...,N 


. When the gradient algorithm converges in, say, k iterations, save the 
state, the control, and the final time, i.e., 


y?, uf; ¿=0,1,...,N; and te 


. Increment the counter r by one and assign 


>w; ¿=0,1,...,N; and >t 


22 


k k 
yi yi, ui 


a 


. Check whether the constraints f = 0 are satisfied throughout the time 
grid. Given a positive real number e3 close to zero if 


N l 


VO filyE ut)? > es 


i=0 j=1 


then go to Step 3. Otherwise, the constraints are satisfied, and the 
values 
y;, up 1=0,1,...,N; and tẹ 


correspond to the minimum. 
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Example 7.5 
Consider the batch reactor problem in Example 7.1 (p.192) subject to the 
following algebraic equality constraint throughout the time interval [0, tr]: 


ur = by + boys 


In this case, 
f = u — biy? — boys = 0 


and the weighting matrix is just an element a” at the r-th outer iteration. 
The corresponding Lagrange multiplier at a grid point c; in the o-interval is 


pi = fia” = (uii — băi — bays,a)a” 


Results 


Using the above algorithm, this problem was solved with bı = —0.1, b2 = 5, 
and the same set of parameters and initial controls as used in Example 7.1. 

With initial controls, the constraint violation in terms of ||g|| was 55.7, 
which reduced and converged to 3.3x107? in 10 outer iterations. At conver- 
gence, the optimal objective functional was —4.45, which corresponds to the 
final product concentration of 4.45 g/cm. The optimal final time reduced 
from 60 to 33.3 min. 

Figure 7.12 plots the initial and final relations between u; and ys. The 
final relation is obtained at the convergence and matches with the imposed 
constraint, f — 0. 
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U1, ul 


¿A constraint: f — 0 ————Uu 
20 F — û 
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ya 


Figure 7.12 The relation between u and y3. The thin solid line at convergence is 
seen to overlap with the imposed constraint 


Numerical Solution of Optimal Control Problems 


213 


The optimal states obtained at the convergence are shown in Figure 7.13. 
Figure 7.14 shows the corresponding optimal controls. 


Figure 7.13 The optimal states when the algebraic constraints are satisfied upon 


convergence in Example 7.5 
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Figure 7.14 The initially guessed and optimal controls corresponding to Fig- 


ure 7.13 
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7.2.4 Integral Equality Constraints 
The objective in this problem is to find the control u that minimizes 


te 


Le ONO) dt 


subject to 


and l integral equality constraints 


te 
F;j(y,u,t)dt = kj, j=1,2,...,l 
0 


As shown in Section 6.4.1 (p. 168), the equivalent problem is to find the min- 
imum of the augmented functional 


te 
M = fa —Aly)dt— u' k 
0 


where p is the vector of time invariant Lagrange multipliers, j4s, correspond- 
ing to the integral equality constraints, L is the Lagrangian given by 


L = H+p'F = F+A'g+pu'F 


and F and k are respective vectors of Fs and kjs. The necessary conditions 
for the minimum of M, and equivalently of J are 


Ă = -Ly y=L1=8 Lu=0 


Alt) =0 Lt) =0 Fdt=k 


o 


along with y(0) = yo. 
When solving this problem using the penalty function method, the integral 
constraints are enforced by specifying 


te 


Hj = ] 5o mac k, Qa, PS 1,2 54 
0 
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where a is a real number greater than unity. With this specification, the 
augmented functional becomes 


te te 2 


l 
m=f H-A'y dt- V; | a-i o 
0 j=l 0 
€ 
penalty function 


The summation term above is the penalty function, which is positive whenever 
any integral equality constraint is violated. 


Example 7.6 
Consider the batch reactor problem in Example 7.1 (p.192) subject to the 
following integral equality constraints: 


te te 


1 1 
zurda and zu 
te te 


0 0 


which specify the average values of the controls over the operation time of the 
reactor. In the transformed o-interval [0, 1], the constraints become 


1 1 
T and f edens 
0 0 


The augmented objective functional is then given by 


penalty function 


where H is provided in Example 7.1. 


Results 


Using the algorithm on p. 210, this problem was solved with îi = 60 cm? /min, 
us = 350 K, and the same set of parameters and initial controls as used in 
Example 7.1. 

With initial controls, the constraint violation in terms of ||g|| was 29.3, 
which reduced and converged to 1.4x107* in nine outer iterations. At con- 
vergence, the optimal objective functional was —7.3, which corresponds to the 
final product concentration of 7.3 g/cm?. The optimal final time in this case 
increased from 60 to 64 min. 

'The optimal states obtained at convergence are shown in Figure 7.15. Fig- 
ure 7.16 shows the corresponding optimal controls. 
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[i 


Figure 7.15 The optimal states when the integral equality constraints are satisfied 
upon convergence in Example 7.6 
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Figure 7.16 The initially guessed and optimal controls corresponding to Fig- 
ure 7.15 
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7.2.5 Algebraic Inequality Constraints 


This problem is similar to that in Section 7.2.3 (p. 209) except that the alge- 
braic equality constraints are replaced with the inequalities 


f(y,u) <0 


Recall from Section 6.3.2 (p. 166) that the necessary conditions for the mini- 
mum are 


y-g  y(0)=y0 A=-Ly | AX(t)-0 
f«0 Lu =0 L(te) =0 p>0, p'L,=0 


where L= F - A'g uf. 

To handle the inequalities, the penalty function method is slightly modified 
as follows. At any time, the multipliers corresponding to the inequalities are 
prescribed as 

u = We 


where e is vector of | elements 


0 if f;t)x0 | 
ej= . 3 —1,2;.. 
1 if f;(t)>0 


Thus, the augmented objective functional 


te 
u- | F+A'(-y+g)+e'We dt 
0 T 


m 


has the penalty function e! Wf, which is positive and enlarges M whenever 
any inequality is violated. The computational algorithm to find the minimum 
is as follows. 


Computational Algorithm 
To determine u and tf at the minimum, 
1. Initialize the counter r — 0. Choose a real number o » 1. 
2. Set the l x l diagonal weighting matrix 
Wc-a'l 


for the (N + 1) grid points of the o-interval to be considered in the 
gradient algorithm. 1 is the l x l identity matrix. 
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3. Apply the gradient algorithm of Section 7.1.4 (p. 191). Use the following 
initial guesses if r > 0: 
uj; i=0,1,...,N and tf 


Note In this case, the augmented objective functional is M whose gra- 
dient in the transformed o-interval [0, 1] is 


VM 2 Loo) Lalo)  .. Lulon) M, | 
where 


L(o; = te F(yi, uj) +A] gly: ui) 


+ [f(y;, u;)] W f(y;, uj); $ = OL... M 
O0 — 


ni 


Using composite Simpson's 1/3 Rule, given that the state equations are 


satisfied, 
1 N N 
M =— Ao+4 A, +2 A +A 


where A; = tF (yi, u) + [f(y;i,uj)] W£f(y;,u;); i=0,1,...,N 


The improvements in u and tf are, respectively, —eLy at each grid point 
and —eM;, where 
1 H 1 N N 
My = do =—— Ho-c-4M Hi; +25 Hic Hy 


dt Nt 
oe 3Nte 1:8,5,.; i—2,4,6,... 


and H= te F(yi,ui, A) Aj g(you4 Ai) ; ¿=0,1,...,N 


The elements of e; (i — 0,1,..., N) are determined as follows: 


0 if f,(t «0 
nec E £050. — ,; 12..5 i20 Lu N 
1 if f;(»0 


4. When the gradient algorithm converges in, say, k iterations, save the 
state, the control, and the final time, i. e., 


k k. 5 k 
y;, uj 1=0,1,...,N; and tj 
5. Increment the counter r by one and assign 


y> yi, ub2w, efe; i-0,L...,N; and >t 
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6. Given a positive real number ez close to zero, if the error 


E-Y [el Eu) > e (7.6) 


then some constraints are violated. Therefore, go to Step 3. 


Otherwise, the constraints f < O are satisfied, and the values 
y;, uj t=0,1,...,N; and tj 


correspond to the minimum. 


Example 7.7 
Consider the batch reactor problem in Example 7.1 (p.192) subject to the 
following inequality constraints throughout the time interval [0, te]: 

U1 2 U1,min and U2,min < U2 < U2,max 


These constraints can be expressed as 


fi —U1 T U1,min 0 
f= fo = |~u2 + U2,min < 0 
fs U2 — U2,max 0 


Results 


Using the above algorithm, this problem was solved with 
Ui min = 50 cm? /mün, U2 min = 300K, Usimar = 400 K 


and the same set of parameters and initial controls used in Example 7.1. 

As observed from Figure 7.3 (p.198), the initial controls do not violate 
the inequality constraints of the present problem. Hence, the initial con- 
straint violation in terms of E in Equation (7.6) was zero. Finally, when the 
penalty function algorithm converged in six outer iterations, the value of E 
was 4.5x107*. The effect of the penalty function was to suppress any viola- 
tion of the constraints during the iterations of the computational algorithm. 
At the convergence, the optimal objective functional was —5.9, which corre- 
sponds to the final product concentration of 5.9 g/cm?. The optimal final 
time decreased slightly from 60 to 58.3 min. 

The optimal states obtained at convergence are shown in Figure 7.17. Fig- 
ure 7.18 shows the corresponding optimal controls, which satisfy the inequality 
constraints. In the absence of the inequality constraints, the optimal controls 
would be those in Figure 7.3 (p. 198), which violate two of the constraints. 
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Figure 7.17 The optimal states when the inequality constraints are satisfied upon 
convergence in Example 7.7 
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Figure 7.18 The initially guessed and optimal controls corresponding to Fig- 
ure 7.17 
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7.2.6 Integral Inequality Constraints 


This problem is similar to that in Section 7.2.4 (p. 214) except that the integral 
equality constraints are replaced with the inequalities 


From Section 6.4.2 (p. 171), the necessary conditions for the minimum are 


y=8 y(0) = yo Al At) =0 
Loc L(t) =0 I<k p20 


and pi! (E — k) = 0 where I is the vector of the integrals js. 
Similar to Section 7.2.5 (p. 217), we prescribe u = We where e is a vector 
of | elements 
0 if L-kjx0 


j=1,2...l 
1 if I; —k; >0 


ej = 


Thus, M has the penalty function e! W(I—k), which is positive and enlarges 

M whenever any integral inequality is violated. 

Computational Algorithm 

The algorithm is the same as that on p. 217 except for the following changes: 
e u, W, and e are uniform across the o-interval. 


e f appearing in the algebraic inequalities is replaced with (I — k). 


Example 7.8 
Consider the batch reactor problem in Example 7.1 (p.192) subject to the 
following integral inequality constraints: 


te te 


1 1 
q uds and = f udt ste 
te te 

0 0 


which specify the upper limits to the average control values over the operation 
time of the reactor. 


Results 


Using the above algorithm, this problem was solved with ü; = 60 cm?/min, 
us = 350 K, and the same set of parameters and initial controls as used 
in Example 7.1. The optimal states obtained at convergence are shown in 
Figure 7.19. Figure 7.20 shows the corresponding optimal controls. 
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t (min) 


Figure 7.19 The optimal states when the integral inequality constraints are satis- 
fied upon convergence in Example 7.8 
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Figure 7.20 The initially guessed and optimal controls corresponding to Fig- 
ure 7.19 


The initial constraint violation in terms of E in Equation (7.6) was 39.1. 
Upon convergence in just one iteration, Æ reduced to zero. At the conver- 
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gence, the optimal objective functional was —6.9, which corresponds to the 
final product concentration of 6.9 g/cm?. The optimal final time in this case 
increased from 60 to 62.9 min. 

[ 


Problems with Fixed Final Time 


The final time is not an optimization parameter in these problems. Thus, 
these problems are straightaway solvable by the algorithms in the previous 
sections using the specified final time and discarding the improvements to it. 


7.3 Shooting Newton-Raphson Method 


Some times the stationarity condition Hu = 0 of an optimal control problem 
can be solved to obtain an explicit expression for u in terms of the state y and 
the costate A. That expression, when substituted into state and costate equa- 
tions, couples them. Thus, state equations become dependent on A and must 
be integrated simultaneously with the costate equations. The simultaneous 
integration constitutes a two point boundary value problem in which 


1. the state is specified at the first boundary, i.e., at the initial time; 


2. the costate is specified at the second boundary, i.e., at the final time; 
and 


3. the state and costate equations depend on state as well as costate vari- 
ables. 


'The shooting Newton-Raphson method enables the solution of this problem. 
With a guessed initial costate, both state and costate equations are integrated 
forward or “shot” to the final time. The discrepancy between the final costate 
obtained in this way and that specified is improved using the Newton-Raphson 
method. 

We explain the shooting Newton-Raphson method with the help of an 
optimal control problem having one state, one control, and fixed final time. 
The objective of the problem is to find the control function u(t) that minimizes 
the functional 


T= | Fiy), u(t) at 
| 
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subject to 


y = gly(t), u(t)), y(0) = vo 
The necessary conditions for the minimum are 
dy dA 


= 0)—yo —=-—H, A(t)-20, and H,=0 
a5 0-w. uu w Alt) — 0, u 
w— >> —_——_—_ : : 
i a stationarity 
state equation and costate equation and condition 
initial condition final condition 


Two Point Boundary Value Problem 


Suppose that it is possible to solve H,, = 0 and obtain an explicit expression, 
u — u(y, A). Utilizing this expression, the state and costate equations for the 
minimum turn into the two point boundary value problem 


dy 


dr > gw X) = 9(u, X. — w(0) = vo 
TF = uliul A = MX). Mt) = e 


h 


Application of the Newton-Raphson Method 


Consider the final costate obtained from the simultaneous forward integra- 
tion of the above equations with A(0) guessed as Ay. The final costate is a 
function of Ap, i.e., A(Ao, tr). Its difference from the specified costate Af is the 
discrepancy function 

f (Ao) = Ao, te) — Ar (7.7) 
which can be zeroed out by iteratively improving Ag using the Newton- 
Raphson method (Section 9.11, p.273). Thus, the improved value Ao next 
is given by 


A0,next = Xo m Of 


where, from Equation (7.7), the derivative 


Of OA 


a a 


Derivative Evaluation 


Observe that the above derivative is the final-time value of the time dependent 
variable 


We will now find the equations governing this variable. 
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Equations Governing 02/00 


Differentiating the state and costate equations with respect to Ao, we get 


0 dy _ 0g 0 dA _ Oh 
Dy dt ^ DA JA dt ^ Or 
, 8 oy _00y 0994 d OA _ Oh Oy , Oh Or 
0 ^ dt OM OyO0X OXON dt OX OyOM  OXOX 
Oy OA 
h £Y (0) = (0) = 1 
where Dro (0) 0 De (0) 


The last two equations arise from differentiating with respect to Ao the initial 
conditions for the state and costate equations, which are, respectively, 


y(0) = yo and A(0)= ào 


Based on the above development, the computational algorithm for the 
shooting Newton-Raphson method is as follows. 


Computational Algorithm 
To determine u at the minimum, 
1. Guess the initial costate, Apo. 


2. Integrate simultaneously to the final time the differential equations for 
the state, costate, Oy/0Ao, and OA/OAo using the respective initial con- 
ditions. 

3. Use 0A/0Ao so obtained at the final time to improve Ag using the 
Newton-Raphson method. 


4. Go to Step 2 until the improvement is negligible. When that happens, 
the state and control are optimal. 


Example 7.9 
Consider an isothermal liquid-phase reaction A —> B in a CSTR in the 
presence of a solid catalyst. The process model is given by 
dy 
dt 
where y is the concentration of species A, u is the time dependent volumetric 
throughput per unit reactor volume, y; is y in the feed, and k is the reaction 
rate coefficient. In a given time tr, it is desired to find the u(t) that minimizes 
the deviation in y and u from the reference condition (ys, Us), i. e., 


=u(y— y) — ky’, y(0)— vo 


te 


r= | (y — ys)? + (u—us)? dt 


0 
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The Hamiltonian for this problem is given by 


H = (y — ys) + (u— uj? +A u(yr — y) — ky? 


'The necessary conditions to be satisfied at the minimum are the process model 
or the state equation with the initial condition, the costate equation with the 
final condition, i. e., 


dA 


Fr —2(y — Ys) +A(u+ 2ky), Alte) =0 


and the stationarity condition 


Hu = 2(u — us) + Alyt- y) = 0 


Two Point Boundary Value Problem 


The last equation yields the expression for the optimal control 


Ma — 
QR S " 
2 
which, when substituted into the state and costate equations, results in the 
two point boundary value problem 


ape c ge h (yc — y) — ky?, y(0) = yo 
dA Aly — 
ae > AU vs) +A MIO y, + 26 , A(t) =0 


The optimal control solution is essentially the solution of the above differen- 
tial equations with boundary conditions at opposite end points of the time 
interval. In order to solve the equations using the shooting Newton—Raphson 
method, we need the derivative state equations to provide Ax. 


Equations for Ax, 


The shooting Newton-Raphson method will use the derivative A), at te to 
improve the guess A(0) = Ao, thereby zeroing out A(tf). We differentiate 
with respect to Ao the latest state and costate equations as well as the initial 
boundary conditions 


y(0)— yo and A(0)— Ao 


A 
assumed 
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to obtain the following derivative equations: 


dya A(yr — y) (yr — y) 
0 = | 3 | —————— si 2k 
P — 3 Us Yro (y — Y) Año + Ayxo 5 Yro 
dA Alys — A 
A = ag t 20579 Lus ok An + (rog) PA tipa A 


along with the respective initial conditions 
Yao (0) =0 and Aro (0) —1 


Computational Algorithm 


1. Set the iteration counter, k = 0. Guess the initial costate, Af. 


2. Integrate simultaneously to the final time the last four differential equa- 
tions for the state, costate, y,,, and Ax, using the respective initial 
conditions. 


3. Improve Af by calculating 


FAS) 


AERE = AE _ 
" 0 Aro (A6; tr) 


4. Check the improvement in Ag. Given a small real number e > 0, if 
| Ast? A| > 
then go to Step 2. Otherwise the state and control provide the minimum. 


Results 


Using the above algorithm, this problem was solved with the parameters listed 
in Table 7.2. The initial guess of A(0) = 0 provided A(t;) = 15.7. Figure 7.21 
shows the optimal state, costate, and control obtained at the convergence, 
which was obtained in nine iterations. 

The minimum objective functional value was 2.4 with final A(tg) of O(10-11). 
The initial and optimal costates are shown in Figure 7.22. 


Table 7.2 Parameters for the problem of Example 7.9 


yo 5g/cm? y  8g/em? yt 10 g/cm? 


us 5 min + k 107? cm?/(g- min) te 1min 
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t (min) 


Figure 7.21 The initial and optimal states and controls in Example 7.9 


1.5 | 


—— — — — —— initial A/ wá 


0.2 | | | | 
0 0.2 0.4 0.6 0.8 1 


t (min) 


Figure 7.22 The initial and optimal costates in Example 7.9. The initial costate 
is in gray 
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7.A Derivation of Steepest Descent Direction 


Consider a continuously differentiable function f = f(x) where x is the vari- 
able vector 
X1 Ta ¿nata Tn 


For a small change 
dx = dz; dz» PM dz, 


the change in f is given by 
df = — da; 
Let the magnitude of dx be denoted by ds. Thus, 


ds = D dz? 


Then the change in f per unit change in ds is 


df n Of dz; n 
dE = 2. m = 2 van = viw 


Oz; ds 
wes” 


Vi wi 


where v is the gradient of f, and w is the unit vector along dx. 


7.A.1 Objective 


Next, consider the problem of minimization of f by changing x by dx of length 
ds along a direction. For a decrease in f due to the change, the rate d f /ds 
has to be negative. The objective is to find the direction or a unit vector w 
that results in the most negative rate d f/ds or, equivalently, minimizes the 
rate, which from the last equation is 


n 
> ViWi 
i=l 


This problem is subject to the constraint that w, being a unit vector, must 
have unit magnitude, i.e., 


kms iie 
i=1 
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Applying the Lagrange Multiplier Rule, the equivalent problem is to find the 
w that minimizes 
JI = Www +A 3 ui-i 
i=1 i=1 


where A is a Lagrange multiplier. Now, at the minimum of J 


dJ = Y (0 +2Awi)dw; +dA `w? -1 = 0 


i=1 ¿=1 
Hence, the necessary conditions for the minimum are 
n 

vj +24w,=0; ¿=1,2,...,n and ) wi-1=0 
i=1 


which yield, respectively, 


a ga _1 ^ a [vi 
u—-—»34xi i—1,2,...,n and As DELL 
Substituting the expression for A in that for w; above, we obtain 


Ui , 
wi = =; 1—1,2,...,n 
MI 


at the minimum. From the above result, the optimal w or w is the unit vector 
opposite to the direction of v, i.e., the gradient of the function f. 


7.A.2 Sufficiency Check 


To ensure that the above result guarantees the minimum, we consider the 
second order Taylor expansion of f with respect to w at Ww, keeping the 
corresponding A fixed at ||v||/2. Thus, for sufficiently small dw, 


J(w + dw, à) — J(w, A) = dJ(w, A) + ssw, A) 


Since dJ(w, A) is zero being the necessary condition, and the constraint K = 0 
implies dK = 0, we obtain 


1 TL TL 

JO + dv, A) — J(w,d) = 3d Jw, A) = AY du?  dAM didu; 
i= ¿=1 

Na ao 


(dK)/2—0 
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Thus, J(w, A), i.e., df/ds corresponding to w, is indeed the minimum. The 
above result shows that df /ds is minimized by 


» U1 v2 Un y 


w=- i Tol Tr 


Ivi dvi III 


which is a unit vector in the direction opposite to the gradient of the func- 
tion f(x). Simply put, the rate of function decrease is maximum along the 
steepest descent direction, i.e., opposite to the gradient of the function. 
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Exercises 


7.1 Simplify the calculation of /^ and JE in the gradient algorithm for Ex- 
ample 7.1 (p. 192). 


7.2 Following Kelley (1962), modify the algorithm for the gradient method 
on p. 191 to determine the final time f£; using the following conditions for the 
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minimum of J with respect to te: 


ar 
d?I OF dyi OF du; 

0 
dg X at, J a 7 


e 


7.3 A CSTR carrying out a first order exothermic reaction is described by 
(Luus and Lapidus, 1967) 


dyi 0241 

ZA = -2y, — a + (yı +01)exp —2 — (yi +0.501)u, 0) = 

d yi — a1 + (yı + a1) exp TIE (yı 1)u, yi(0)= y1,0 
dya 0241 

=== — — ——— , 0 = 

d = ta (y2 + a1) exp EET y2(0) = y2,0 


where yı and ya are, respectively, the deviations of temperature and concen- 
tration from the steady state values. The control u is related to the coolant 
flow rate. Find the u that minimizes state deviations and control action given 
by 


ig 


Pe fe + yi + azu’) dz 
0 
for the following set of parameters: 


y1,0 = 0.05 y2,0 = 0 te = 0.78 
ay = 0.5 ag = 25 a3 = 0.1 
7.4 Repeat Exercise 7.3 with the following additional constraints: 


0.4 <us 1.4 


7.5 Consider the process of production planning described by 


cu =u, y(0)=0 
where y is the product inventory, and u is the production rate. It is desired 
to find the u that minimizes the cost functional 
te 
I= [law + azy) dt 


0 


subject to the following constraints: 


y(t) = ye and u>0 


Numerical Solution of Optimal Control Problems 233 


Using a suitable computational algorithm, solve the above problem for 
a, =4=Yy=tp=1 


Compare the optimal u with the analytical solution (Kamien and Schwartz, 
1991) given by 


u = 


Gai) y YE irl) 


4a4 te 


7.6 The integral equality constraints of the optimal control problem in Sec- 
tion 7.2.4 (p.214) may be transformed into differential equations. Using this 
approach, 


a. derive the necessary conditions for the minimum, 


b. formulate an algorithm using the penalty function method to find the 
minimum, and 


c. solve Example 7.6 (p. 215). 
7.7 Repeat Exercise 7.6 to solve Example 7.8 (p. 221). 


7.8 Solve Example 7.8 (p.221) in the presence of the following additional 
constraint: 
ua < 400 K 


throughout the time interval [0, tr]. 


Taylor & Francis 
Taylor & Francis Group 


http://taylorandfrancis.com 


Chapter 8 


Optimal Periodic Control 


Optimal periodic control involves a periodic process, which is characterized 
by a repetition of its state over a fixed time period. Examples from nature 
include the circadian rhythm of the core body temperature of mammals and 
the cycle of seasons. Man-made processes are run periodically by enforcing 
periodic control inputs such as periodic feed rate to a chemical reactor or 
cyclical injection of steam to heavy oil reservoirs inside the earth's crust. The 
motivation is to obtain performance that would be better than that under 
optimal steady state conditions. 

In this chapter, we first describe how to solve an optimal periodic control 
problem. Next, we derive the pi criterion to determine whether better periodic 
operation is possible in the vicinity of an optimal steady state operation. 


8.1 Optimality of Periodic Controls 


Consider the objective to minimize the functional 


te 


I= J Fly(0),u(t)] dt 


0 


where tf > 0 is the time period of the control u, and the state is governed by 


y = gly(t), u(t)] 
with the periodicity condition 


y (0) = y (tr) 


The above problem is equivalent to minimizing the augmented functional 
te 
Jal F+A'(—y +g) dt 
0 
subject to y(0) = y (tg). 


235 
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8.1.1 Necessary Conditions 
Similar to Section 6.1.1 (p. 153), the variation of J is 


tf 

ie I Hori oyan 6A 4- Hl du dt + XT(0)5y(0) 
0 
— A (tc) dye + H(te)dte 


Note that the above equation has one additional term, '(0) dy(0), when 
compared to Equation (6.9) on p. 156. The reason is that 6y(0) in the present 
problem is not zero since y(0) is not fixed but equal to y(tr). This periodicity 
condition for the general case of unspecified t; means that y(0) is equal to the 
final state yr. As a result 
óy(0) = óyr 

Considering this fact, the necessary conditions for the minimum of J, and 
equivalently of I, are 


Á=-—H, y =Hx1=8 Hu =0 
A(0) = A(tr) y (0) = y (tr) H(t) =0 


From now on, we will use 7 instead of tf in the periodic problems. 


Example 8.1 
Consider the periodic operation of the batch reactor in Example 2.10 (p. 45) 
with the objective to minimize 


x 
E 
f=- ka“ dt, k=k == 
fe z j o exp RT 
0 
subject to 
i=-—akx", x(0)=x(7) 
using the temperature T(t) as the control over a fixed time period 7. 
The Hamiltonian for this problem is 
H = —ckz^ — Maka" 
Thus, for J to be minimum, the necessary conditions are as follows: 


t= aka", x(0) = z(T) 
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Note that if 7 is not fixed, then we have the additional necessary condition 


H(t) = — ckr“ + akz? =0 


T 


Example 8.2 


Consider the CSTR described in Example 6.7 (p.164) but under periodic 
operation in which the objective is to minimize 


where 7 is a fixed time period. The state is governed by 


d = u(y — y1) — kyiya, yi(0) = (7) 


Yo = U2 — uiya; ya(0) = ye(T) 


as well as the equality constraint 


1 T 
[uaa Or T 
m 

0 


which requires the average CSTR throughput to be some desired value a. The 
Lagrangian for this problem [see Equation (6.19), p. 169] is given by 


L= (yi — Yi)? + (ui — UR)? + Arr (Ye — y1) — kis] 


2 
=1 


2 


H Alu — ury2] + uui 


Note that the Lagrange multipliers A, and A2 are time dependent, while the 
multiplier u is time invariant. The necessary conditions for the minimum of 
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I are as follows: 


ui(yr — y1) — ky1ya yı yı 


u2 — U1Y2 Y2 Y2 


t=0 T 
M— 0 — MH 


Aa 2(Y2 — yg) + ALky1 + A2 A2 Az 


ÎS ——— 
—Ly 


Lu, 


ui) + AY(yt — 91) — A2ye + pl 0 


Luo 


[a= ua = 
0 


Inequality Constraint 


2(u2 = us) + Aa 0 


T T 
A E — yi) + Ai (u1 + ky2) à ^ 


If we do not want the CSTR throughput to surpass some maximum value a, 
then the equality constraint in the above problem has to be replaced with the 
inequality 


T T 


1 
= usa or fatar 
E 


0 0 


In this case, the necessary conditions for the minimum are given by (see 
Section 6.4.2, p. 171) 


2 
e the ones above except f Lu dt = ar, which is replaced with 
0 
34 T 


| to f < ar 


0 0 
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e «> 0, and the complimentary slackness condition 


T 


u T =0 
0 


8.2 Solution Methods 


The solution of a optimal periodic control problem requires the integration 
of state and costate equations, both subject to periodicity conditions. Other 
than this integration aspect, the solution methods for optimal periodic control 
problems are similar to those for non-periodic problems. Therefore, we will fo- 
cus on the methods to integrate state and costate equations under periodicity 
conditions. 

A periodicity condition implies that the initial and final values of a state 
(or costate) variable are equal to a single value. Thus, in a optimal periodic 
control problem, the set of state as well as costate equations poses a two 
point boundary value problem. Either successive substitution or the shooting 
Newton-Raphson method may be used to integrate the periodic state and 
costate equations. 


8.2.1 Successive Substitution Method 


This is a simple but slow method in which a set of state (or costate) equations 
is integrated assuming the initial conditions. The final conditions obtained 
from integration are then substituted for the initial conditions in the next 
round of integration. This procedure is repeated until the initial and final 
conditions match. 


8.2.2 Shooting Newton-Raphson Method 


This method was introduced in Section 7.3 (p. 223) to solve a two point bound- 
ary value problem. We illustrate this method to solve a set of state equations 
with periodicity conditions. They require zeroing out the discrepancy function 


F(yo) = vol =- Yo, 
—— initial state 


final state vector 


vector 
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where we use yo to denote the initial state vector y(0). Starting with some 
guess, y(0) is improved iteratively using the Newton-Raphson method, i. e., 


Yo,next = Yo — [fy];  f(yo) (8.1) 


where [fy,], is the Jacobian 


du] dy dy 
OY1,0 Oy2,0 XY Oyn.o 
us Ius -1 n Lie 
Y1,0 Y2,0 Yn,0 (8.2) 
0Yn 0Yn OYn EE 
0Y1,0 Oy2,0 | Oyn,o " 


which is obtained by differentiating f(yo) with respect to yo, i.e., y(0). 


Evaluation of the Jacobian 


The above Jacobian is the matrix fy, evaluated at the final time r. The time 
dependent elements of fy, are governed by differential equations that arise 
from the state equations. For example, differentiating the ¿-th state equation 
and the initial condition, 


yi = gi and yi(0) = vio 


with respect to y;,9 yields the following differential equation: 


d Oy -y 28 Yk Oy; _ 1 if i=j 
dt ÓOyjo £— Oy ui. Di vum 0 if i=j 


whose integration provides the desired value of Oy;/Oy;,9 at the final time. 
The computational algorithm to solve state equations with periodicity con- 
ditions is as follows. 


Computational Algorithm 
1. Assume the initial conditions for the state equations. 


2. Using the initial conditions, integrate simultaneously the differential 
equations for state variables and 0y;/Oy;,08, which constitute fy,- 


3. Improve the initial state using 
YO,next = Yo — [fy], £(yo) 


Go to Step 2 until the improvement is negligible. 
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4. Repeat the above steps for costate equations. 


Example 8.3 
Consider a CSTR under periodic operation governed by the state equations 
dyi 
a > ue) RR w(0-mu() 
d 
ES = u» — Up, y2(0) = ya(r) 


where y; and ya are concentrations of the reactant and catalyst, R = koy?y2 
is the rate of production, yr is the reactant concentration in the feed, and u1 
and ug are, respectively, the volumetric throughput and catalyst mass flow 
rate per unit reactor volume. The controls u; and u have the time period 7. 
The objective is to find the controls and the time period that maximize the 
average R, i.e., minimize 


T 


1 
I= -E | nout dt 
T 
0 


Transformation to Fixed o-interval 


We begin by transforming the problem to the fixed o-interval [0, 1], as was 
done in Section 7.1.3.1 (p. 187). The equivalent transformed problem on this 
interval is to find the controls and the time period that minimize 


subject to the state equations 


dy 

de =r ui(yr— yi) — hoyiya > yi(0) = (1) 

d 

T =T UW— wy , ya(0) = ya(1) 
Oo 


where the controls u¡ (7) and u2(0) have the time period of unity. The Hamil- 
tonian for this problem is 


H=-—koyiya +7 Ar ur(ye— y1) — koyiya + A2(u2 — u1ya) 


In terms of H, the augmented objective functional is given by 


1 
= dyi dya 
de] Heap To do 
0 
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The necessary conditions for the minimum are 
1. the state equations in the o-interval 


2. the costate equations 


dA 

E = 2koy1y2 + TAL (uz + 2koyiye2), Ai (0) = A1(1) 
di> 2 2 _ 

do ^ "wi + Toys + Agta), A2(0) = A2(1) 


3. the stationarity conditions with respect to the controls u and uz, i.e., 


Ha, - Ai(yr — y1) 0 
Hy, A 0 


4. the stationarity with respect to the time period 7, i.e., 


1 
=f RCRUM dines (8.3) 
0 


T 


Satisfaction of Periodicity Conditions 


In order to integrate the state and costate equations satisfying the periodicity 
conditions, we need the respective derivative differential equations for the 
shooting Newton-Raphson method. 


Derivative State Equations 

For each i = (1,2), taking the partial derivatives of 
1. the state equation for y; and 
2. the initial condition y;(0) = y; o 


with respect to all guessed initial states — y1 o and y»,o — yields the following 
derivative state equations: 


d dp yı 2 Oyo dm 


y 
=T —(ui + 2k ——-— : 1 

do Owyio (u YY Gp 981 OY1,0 OY1,0 
d Og Oy 2 Oy» Oy: 
— —T —(u + 2k — — — koyi—— , 0)=0 
do yao (u VAL on OY2,0 OY2,0 ) 

10) 
A NEN V? (0) — 0 
do OY1,0 OY1,0 OY1,0 

19) 
ER Oy = fai Oye | y2 0)—1 
de yao 0Y2.0 OY2,0 
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Derivative Costate Equations 


Similar differentiation of the costate equations yields the following derivative 
costate equations: 


d OA OA DA 


ds BX A 5x0) - 1 
i vem mud Nana ge As Mes 
do ze e hug a i Aus Nest 


The integration of the derivative state and costate equations provides the 
derivative values at the endpoint o = 1. These values are needed in the 
Jacobian of the Newton-Raphson method [see Equations (8.1) and (8.2)]. 

Following is the computational algorithm of the shooting Newton-Raphson 
method to solve the optimal periodic control problem. 


Computational Algorithm 


1. Set the iteration counter k — 0. Assume 7* and an even N. Obtain the 
fixed o-grid of (N + 1) equi-spaced grid points 


oo(= 0), 01; 02, seg ON-1; on(= 1) 


At each grid point, assume a value for controls, initial state, and initial 
costate as follows: 


Kg, 
uf =u'(0,) = pita , $50,1,...,(N - 1; and uk =u 
u5 (ai) 
k k 
yi (00) Aj (00) 
yb =y*(o0) = |" "|, Aj=M00)= | 
y3(co) 2(00) 


2. Integrate state equations as follows: 


2.a Set the counter s = 0. Assume the initial state 


z 
s— „k 
Yo = Yio Y2,0 
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2.b Integrate forward from o = 0 to 1, the state equations along with 


derivative state equations using the control functions ufs. 


2.c Improve the initial state by applying Equations (8.1) and (8.2), 


i.e., 
-1 
Oy 0y1 
+1 —1 1) — y? 
Yio = Yio = OY1,0 Oy2,0 n) Ui, 
+1 
V5.0 92.0 dy dy E 
ES EL em E) E -1 y2(1) — v3,o 
yz yá Y1,0 Y2,0 o=1 
[fyo]; ave) 


2.d Calculate the vector of change in yo, 


z 
e= (ib — Yi o) (uio — Y o) 


Given a small real number e > 0, if 


either |e|] or f(y°) > e 


then assign yt — y, increment s by one, and go to Step 2b. 


Otherwise, the integration of the state equations is complete. 
3. Save the values of state variables at the grid points. 


yr = y" (oi) = ] ’ 1=0,1,...,N 


4. Evaluate the objective functional. For example, using composite Simp- 
son's 1/3 Rule, 


N N 
1 
Mala Fo + 4 y F;¡+2 5 F; + Fy 
1,3,5,... 1=2,4,6,... 


where 
F; = —ko(yf i) ya; 1=0,1,...,N 


5. Check the improvement in I* for k > 0. Given a tolerable error & > 0, 
if 
[PIPE <e 
then go to Step 12. 


6. Integrate costate equations as follows: 
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6.a Set the counter s = 0. Assume the initial costate 
; T 
Ay = Ao Año 
6.b Integrate forward from o = 0 to 1, the costate equations along 


with the derivative costate equations using the control functions 
u's and the state variables y^s. 


6.c Improve the initial costate by applying Equations (8.1) and (8.2), 


i.e., 
-1 
dA OM 
sr S XE 1 == "i 
ADT = Alo _ OA1,0 Od2,0 Aa (1) 1,0 
Mo o 9) 9)» 
: i _—= —1 A2(1) — AS 
TE um DA Od2,0 e 2(1) 7 Xo 
0 0 NI os 
I]. "ees 


6.d Calculate the vector of change in Ao, 


e OI — Mo) (AS) — 3.0) 
Given a small real number e > 0, if 
either |el or ||f(Ao)|| > e 


then assign ADU — Aj, increment s by one, and go to Step 6b. 
Otherwise, the integration of the costate equations is complete. 


7. Save the values of costate variables at the grid points. 


IC 
AF = A" (gi) = (e) i=0,1,...,N 
3 (01) 


8. Evaluate the gradient by calculating the partial derivatives 


Ay k ut AP 
HE = Hie) = riw "s i=0,1,...,N 
Hua, (yE, u}, AF) 


N N 
1 À 
and Ji=— Ap+4 XO A}+2 Y AR+ A 


3N 1,3,5,... i—2,4,6,... 


H(yk, uf, AP) - F(yF, ub, AF). 


where Af = 7 : 


i=0,1,...,N 


T 
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Check the magnitude of the gradient. Given a small positive real number 
£2, if the norm of the gradient 


N 2 2 


mm H (y$, un AT) E JF < ez 


i=0 j=1 
then go to Step 12. 


9. Improve control functions by calculating 


a" = uf — HE 


u,i 


i=0,1,...,N 
h $a MEI l b . . d . . I* 
where € isa positive real number causing maximum reduction in : 


10. Improve the time period using 


qu — 7f — p 


11. Repeat calculations Step 2 onward after assigning 


k+1 


k+1 k k j 
T; —T; and uj -—uj;, 21=0,1,...,N 


and incrementing k by one. 


12. Terminate the algorithm with the following result: 


The optimal objective functional value is IF. 

The optimal control a(t) is represented by u^, i = 0,1,..., N. 
The optimal time period is 7^. 

The optimal state ¥(t) is represented by y^, i = 0,1,..., N. 


Results 


Using the above algorithm, the optimal periodic control problem was solved 
for the parameters listed in Table 8.1. The objective is to find two control 
functions and the time period that maximize the average reaction rate. 


Table 8.1 Parameters for the problem of Example 8.3 


yr 5 g/cm” 
ko 6 x 107 (cm? /g)? /min 
N 40 


The initial and optimal states are plotted in Figure 8.1. The corresponding 
controls are shown in Figure 8.2. 
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Figure 8.1 The initial and optimal states for Example 8.3 
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Figure 8.2 The initially guessed and optimal controls for Example 8.3 


With the initially guessed time period of 6 min, the initial controls provided 
I = —7.8 x 107?, which corresponds to the average production rate of 7.8 x 
107? g/min. 


The application of the algorithm minimized J, i. e., maximized the product 
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concentration to 0.12 g/min. The optimal time period increased to 6.8 min. 
Figure 8.3 shows the convergence of J to the minimum. 


iteration 


Figure 8.3 The objective functional versus iteration for Example 8.3 


8.3 Pi Criterion 


The pi criterion is a sufficient condition for the existence of a periodic solu- 
tion that is better than the neighboring optimal steady state solution of an 
optimal periodic control problem. Using the criterion, we would like to know, 
for example, whether the time-averaged product concentration in a periodic 
process can be more than what the optimal steady state operation can pro- 
vide. In other words, we would like to check if oscillating the optimal steady 
state control with some frequency and time period improves on the steady 
state solution. 

We derive the criterion for a general periodic problem of finding the mini- 
mum of the functional 
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subject to the following constraints: 


y =s(y,u), y(0)=y(7) (8.5) 
= [re u) dt =0 (8.6) 
1 
— | w(y,u)dt <0 (8.7) 
[e 


1 
0 
1 f ; 
0 
1 f : 
Ly) => fuy, )dt t=nt+1,n+2,...,p (8.9) 
T 
0 
1 f , 
0 
we introduce the following functional 
p q 
J(y,u, A, p, v) = = (y, u A Ll (y, u tM pi Li (y, u) + 5 vili(y, u) 
i=n+1 i=p+1 


incorporating the multipliers A;s, pis, and v;s. Now in terms of the Lagrangian 


p 
L(x, u, A, u,v) = M Aigi(y,u + y pio (y, u) + 5 vjw;(y,u) 


i=n+1 i=p+1 


the functional J can be written as 


1 
Joa Auer) == | LA) at 
0 
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J at Optimal Steady State 


Under the steady state, both u and y are time independent. Then the above 
problem reduces to the minimization of 


I =F(y,u) 


subject to 


se 
Il 
n 
= 
e 
i 
© 


= fwya = w(y,u) < 0 
0 


Let the pair (y, u) denote the optimal steady state solution with the corre- 
sponding multipliers A, ji, and P. The necessary conditions for this solution 
are provided in Appendix 8.A (p. 260). Since this solution minimizes J, sat- 
isfying the state equation and the constraints, 

" 

- 1 S 
JJa A 1,0) = Tu) = 2 | La X po) t 
T 
0 


In deriving the above equation, we have considered the fact that all equality 
constraints are satisfied at the optimal steady state, and the multipliers are 
zero for inactive inequality constraints (see Section 4.5, p. 109). 


Further Reduction of / 


We ultimately need to determine whether in the vicnity of the optimal steady 
state pair (y, u), the objective functional J could reduce further for some 
admissible pair of state and corresponding control, (y, u), where 


y=ytoy and u=u+du 
The pair (y, à), as well as the control ü, is called proper. 
Assumptions 
We assume that 
1. the Lagrange Multiplier Rule is applicable 


2. F is twice differentiable, and the derivatives of gi, vi, and w; are con- 
tinuous in the neighborhood of the optimal steady state solution (y, u) 
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The first assumption implies that the constraint qualification or normality 
condition (see Section 4.3.2, p.96) is satisfied. It ensures that an infinite 
number of solution pairs exist near the optimal steady state solution pair 
(y, a). This pair, as well as ū, is called normal when the normality condition 
is satisfied. 

Next, we retain the multipliers at the optimal steady state and obtain 
the change in J from its optimal steady state value, i.e., J(y, u, A, p, v) — 
J(y,u, A, p, v). This change is given by 

n p qo 
i=1 


i=n+1 i=p+1 
= = fa-Da = -[eneiüna 
T 
0 0 


where qo is the number of inequalities satisfied as equalities and the overbar 
as in I indicates the steady state value. In the last equation, we have used 
Equation (2.33) on p.52 and considered a sufficiently small change along dy 
and du at y and u, respectively. Note that the multipliers are zero for inactive 
inequalities, which are satisfied as strict inequalities. Also, all /;s are zero for 
the qo equalities since they are satisfied by the steady state solution. 

From the result proved in Appendix 8.B (p. 261) for the normal steady state 
pair (y, ü), we have 


Ij(y,u) = ôl; = ôl; (y, ū; ôy, u); j= 1,2,...,n +p + qo (8.12) 
Therefore, from the last two equations, we obtain 

p RSS a Ji 5D+182L de (813) 

i=1 i=n+1 i=p+1 : 


In terms of the variations of L and all I;s obtained from their definitions in 
Equations (8.11) and (8.8)-(8.10), we have 


puer PY Mtn + > 100, + > Diw; dt 
0 


i=n+1 i=p+1 


SL 4 5 juól,; + > nôl, 


i=n+1 i=p+1 


Substituting the last equation in Equation (8.13), we obtain 


I-I- ~ | 6Fat+ x ] ^ia (8.14) 
0 
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We shall return to the above equation after coming up with an admissible 
pair (y, u), or equivalently, an admissible variational pair (dy, du). 


Admissibility Criteria 


The pair (y, u) is admissible if and only if it satisfies the constraints — Equa- 
tions (8.5) and (8.6) and Inequality (8.7). 

If (y, u) is admissible, then the related variational pair (ôy, ôu) is called 
admissible as well. This pair must satisfy some constraints, which are de- 
termined as follows. For (y,u) the constraints in terms of sufficiently small 
variations dy and du near (y, à) become 


y +0y =8+8,0y +E,0u, —y(0)--óy(0) = y(7) + óy(7) 


1f 1f 
zy + vudu) ar = o, = | (w+ wydy + sud) dt < 0 
0 0 


The above constraints respectively simplify to 


dy = 8y0y +8,0u, dy (0) = dy(r) (8.15) 
z Jr + Vudu) dt = 0 (8.16) 
m 
0 
Lf _ 
= ns + waóu)dt < 0 (8.17) 
» 


0 


which are the equations that govern dy and du. Note that the above simpli- 
fication is due to 

"P " lf. 1f. 

y-g y(0)=y[(7) —]vdt-0 and — | wdt<0 

T T 
0 0 

which are obviously satisfied by the optimal steady state pair (y,u). As a 
consequence, the admissible variation pair (dy, du) is constrained by Equa- 
tions (8.15) and (8.16) and Inequality (8.17). 


Admissible (dy, du) at the Optimal Steady State 
We propose the following control variation 
óu(t) = 2U cos(wt) where w = 2a/r (8.18) 


at the optimal steady state control ü. After showing that the corresponding 
variational pair (dy, du) is admissible, we will derive the sufficient condition 
for the resulting objective functional I(y, u) to be lower than the steady state 
I(y,ü) or I. 
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We now apply the admissibility criteria of the previous section. For the 
variation pair (dy, óu) to be admissible, du(t) should generate a state variation 
dy governed by Equation (8.15). Moreover, the variation pair must satisfy the 
remaining constraints, i.e., Equation (8.16) and Inequality (8.17). 

In the following treatment, we use the Fourier transform, the details of 
which are provided in Appendix 8.C (p. 264). Using Euler’s formula 


e“? = cosg + isin z 
for x = wt, the control variation can be expressed as 
óu(t) = U e îi + ett (8.19) 
From the definition of the Fourier transform and its inverse 
óu(t) = óu(- ne *** + fujet 


where 
óu(-1) —óu() = U 


are the Fourier coefficients of the proposed control variation óu. 


Satisfaction of Equation (8.15) 
Taking the Fourier transform on both sides of the differential equation given 
by Equation (8.15), we obtain 
dy (k) = (ikw)ðy (k) = By dy(k) + gyÓu(s) 
Upon rearranging the transformed equation, we get 


yk) = (ikul — g,)~*By Sur) (8.20) 
N ë aa 
G(ikw) 
For the proposed du, there are two Fourier coefficients du(k=+1), each being 
equal to U. Making these substitutions in the above equation and taking the 
inverse Fourier transform yields the corresponding 


dy = Gdu-ye ^. + Gume = GUe** + GUe™ (8.21) 


where G = G(iw) and G = G(—iw). Observe that óy(0) = óy(r). Hence, 
the variational pair (dy, du) satisfies Equation (8.15). 


Satisfaction of Remaining Constraints 


The variational pair (dy, du) should also satisfy Equation (8.16) and Inequal- 
ity (8.17) in order to be admissible. The left-hand side of an i-th constraint 
from Equation (8.16) or Inequality (8.17) is 


= 1 f . 
an => | ay dy +4,0u dt; ¿=(n+1),(n+2),...,q (8.22) 
0 
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where a is, respectively, v; or w;. Substituting for du and dy from Equa- 
tions (8.19) and (8.21), we get 


ôl; == Ja GUZ 4 0Ug “aU e™™t + et )dt=0 
0 


after integrating and applying the limits. Basically, we have applied the result 


dE S (8.23) 


F 
[ec dt = Em gphi(27/7) 

kiw 0 
0 


for a constant c and a non-zero integer k. Hence, the pair (6y,6u) satisfies 
Equation (8.16) and Inequality (8.17) as well, and thus is admissible. 


Sufficient Condition or Pi Criterion 


We now utilize Equation (8.14), which provides the change in / from its 
optimal steady state value. For the present problem, ôF is zero since (y, ū) 
is optimal. Expanding 92L in Equation (8.14), we obtain 


= 1 f E = = de 
Peran J dy Ly, + ou" Lju óy + dy' Lyu + ôu' Luu du dt (8.24) 
T 
0 


=U G Lg i iLO G ts U (8.25) 
E 
II(w) 


where II(w) is an (m x m) matrix known as the pi matrix. Appendix 8.D 
(p. 265) provides the derivation of Equation (8.25). For I to be lower than J, 
obviously 

I-I = U'II(w)U < 0 


or, in other words, the pi matrix has to be negative-definite. This result is 
known as the pi criterion. 


Example 8.4 

Consider a second order reaction A —> B of rate coefficient k carried out in an 
isothermal CSTR. Assuming no volume change of mixing, the concentrations 
x, and 22, respectively, of the reactant A and the product B are governed by 
the state equations 


Ly = us (ur = 21) = ka? 


La = ka? — U2T2 


where the two controls are 
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1. u1, the time dependent concentration of A fed to the reactor and 
2. u», the volumetric flow rate in and out of the reactor 


both per unit reactor volume. For the periodic operation of time period 7, 
the objective is to maximize the average concentration of the product B, i.e., 


to find the minimum of " 


1 
15-2 f vit 
T 

0 


subject to the following two equality constraints: 


1. The average mass of A fed to the reactor should be some value a, i.e., 


T T 


1 1 
= fumdt=a or = f (us - aat o 
T F — a 


0 0 V1 


2. The average volumetric rate through the CSTR should be some value 


D, i.e., 


T T 


1 f wsae= 6 " za p)at=0 


ES == => 
0 0 va 


The Lagrangian for this problem is given by 


L = -r + Aj [us (ui = 21) = ka?) + Aa [ka? — U2%2 
- 7 Ne —M'. — M 
F gı 92 
+ m (uru — o) + a (us — B) 
A EE — == 
vi va 


Let the optimal state and control under steady state conditions be 
x= and u= 


Sufficient Condition for (x,ú) to Be Proper 


The optimal steady state pair (x, u) is proper if there exists a neighboring pair 
(x,u) that provides a lower objective functional value J than I, which is given 
by (x,u). Assuming that the pair (x, u) is normal, the sufficient condition 
for its properness is that the right-hand side of Equation (8.25) should be less 
than zero, i.e., the pi criterion 


UT C £641 C20 LA U < 0 
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where U = Ui Ur T is the amplitude of ĝu = Sua dug E which is 
the periodic control variation given by Equation (8.18) for some frequency 
w = 27/7. Both U and w need to be specified. The matrices in the above 
inequality are as follows. 

The matrices G and G are given by 


G=G(iw) and G = G(-iw) 


where for k = +1, we have from Equation (8.20) 
-1 
ik 0 —U2 — 2kx 0 
Glikw) = 4 | a Ni x 
0 ikw 2kx1 —U9 
—À O —á 
I(ikw) Bx 


Lyx, Lxu, and Luu are the matrices of second order partial derivatives 


| -2A1k+2%k 0 Lo 0 =A 
i 0 0 TT du 45x. 
Das i 0 Aic fit 
A+ fa 0 


which arise from the vectors of partial derivatives 
AM (us = 2kx1) + 22d kx1 
—1 — AQua 


Ly = 


Ayu2 + pua 
Ai (ur — 21) — A222 + 141 + H2 


8.4 Pi Criterion with Control Constraints 


As long as the optimal steady state controls are normal and lie within but 
not at the boundaries of the set of admissible control values, the negative- 
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definiteness of the pi matrix is the sufficient condition for the neighboring 
controls to be optimally better. This condition does not hold if an optimal 
steady state control ü; lies at its boundary 4; max, as shown in Figure 8.4. In 
this case, du; given by Equation (8.19) becomes invalid since u; = ü; + du; 
trespasses the boundary for any non-zero Up. As shown in the figure, this 
situation can be avoided by adding a suitable constant v; to the right-hand 
side of Equation (8.19) such that u; lies within the boundary. 


invalid ui = Ui + du; 


valid ui = —Aui + ti + dui 
—— 


vi 


0 t T 


Figure 8.4 Controls near the upper limit wimax 


However, we need to derive the sufficient condition again. Let the control 
variation be given by 


du = V + 2U cos(ut) = V +U e biet, w= = (8.26) 


where an element V; of V is 
e zero if U; is within and away from the control boundary, and 


e non-zero if u; is at the control boundary — the value of V; being such 
that u; is within the control boundary. 


From the definition of the Fourier transform and its inverse (see Appendix 8.C, 
p. 264) 
óu(t) = du(-1)e ** + óu(o) + óu()e'^* 


where 
duo) = V and óu(-1 = 0u(1) = U 


are the Fourier coefficients of the proposed control variation du. 
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Satisfaction of Equation (8.15) 


Substituting the above Fourier coefficients of du in Equation (8.20), we obtain 
dy = GUe-iut + GV + GUe (8.27) 


where 
G=G(0) = -8y'Bu 


Observe that dy so obtained satisfies Equation (8.15) as well as the periodicity 
condition dy(0) = dy(r). 


Satisfaction of Remaining Constraints 


To be admissible, the variational pair (dy, du) thus obtained should satisfy 
the constraints given by Equation (8.16) or Inequality (8.17). Thus, the 
corresponding ôl; given by Equation (8.22) should be zero for the equality 
constraints and non-positive for the inequality constraints. In the foregoing 
treatment, we assume that these constraints are satisfied so that (dy, du) is ad- 
missible. In other words, the corresponding pair (y, u) satisfies Equation (8.6) 
and Inequality (8.7) on p. 249. 


Sufficient Condition or Pi Criterion with Control Constraints 


Switching to Equation (8.14) on p. 251 for the change in J from its optimal 
steady state value J, we should appreciate that dF is not necessarily zero in 
the presence of control constraints. For example, if all controls under the 
optimal steady state lie within the control boundaries except t,, which is at 
its upper limit 41 max, then for sufficiently small variations (dy, ôu) for which 
521 tends to zero, Equation (8.14) yields 


"ELS i s a 17 
I-I = ZEIT = [5 Fu, 0u¡ dt = = | Padua > 0 
T T =] T 
0 0 > 0 


because of the following reasons: 


e Inside the control boundaries, Fa, for i > 1 must be zero as 6u; can 
be either positive or negative. Otherwise, for a non-zero Ps one could 
choose a suitable ĝu; to contradict the above inequality, i.e., the mini- 
mality of I. 


e J — Ī > 0 since J is the minimum. 


The term Fu, is not necessarily zero. With à; at its upper limit, du; can only 
be negative. Its coefficient Fu, thus could be either zero or negative for I to 
be minimum. Consequently, dF is not necessarily zero. 
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Properness of (y, u) 
) 


For the pair (y, à) to be proper, I — I < 0. From Equation (8.14) 


Then (I — I) is less than zero if 
e either ôI = 0 and 62] is less than zero, or 
e both ôI and ó?I are less than zero. 


Upon expanding ôF and 62L, we obtain 
_ 1 _ _ 
I-I= = [iy + Fuðu) dt + 
T 
0 


if - " : : 
= / dy Lyy + dul Ly, dy + dy" Lyu + óu! Luu du dt 
- 

0 


Integrating the right-hand side of the above equation after substituting for 
óu and óy, respectively, from Equations (8.26) and (8.27), we finally get 


= A V'TI(0)V 
I-I=(F,G+EJV+ AY + UTII(u)U 
E EA 
ôI 527 


Hence the sufficient condition for (y, u) to be proper is that 


n V'TI(0)V 


(FyG+Fu)V <0 and 3 +U!TI(w)U <0 (8.28) 


where the matrices II(0) and II(w) are given by, respectively, 


II(0) =G"LyyG+11,6 +6" Lyu + Luu and 
HU) SG Ore GFE Latim 
Example 8.5 


Consider Example 8.4 (p.254) with the additional control inequality con- 
straint u < Umax- 

The sufficient conditions for properness of the optimal steady state pair 
(x, u) stay the same as long as u is less than Umax. However, if a Ui = Ui max» 
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then an appropriate control variation is given by Equation (8.26) in which the 
T _ 
vector V= Vi Və  ensuresthatu=u-+6u does not exceed Umax- 
Hence, V; is 
e zero if the corresponding Ui < tj max, OT 
e some suitable value to ensure that the neighboring perturbed control 
Ui < Ui max if ui = Ui, max: 


Thus, provided that the pair (y, u) satisfies the constraints of the problem, 
the sufficient conditions for properness, from Inequalities (8.28), are 


= v'II(0)V 
(F.G+Fy)V €0 and Y TOM ) + UTII(u)U <0 
where 
0 
Fx = and Fu = 
—1 
—1 
A bi ati | E 
2kzi —U2 0 —X2 
Bx Bu 
TI) = G' LyxG + Lla G GI Lyu + Luu 


Hi = La +A LEAC Lyu +t Luu 


The matrices G, G, | pone and Luu are the same as derived earlier in 
Example 8.4. 


8.A Necessary Conditions for Optimal Steady State 


We present the necessary conditions for the steady state minimum in the 
problem described by Equations (8.4)-(8.7) on p. 248. 

Let u be the optimal control under steady state with the corresponding state 
y, and multipliers A, j1, and v; all of which are time invariant. According to 
the John Multiplier Theorem (Section 4.5.1, p. 113), the necessary conditions 
for the minimum of I subject to the equality and inequality constraints are 
as follows: 
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D0;,>0 when w;(y,u) 


=0 
fori=p+1,p+2,..., 
and 7; 20 when wj;(y,u)«0 idi B T 


fcre 
< < 
c c 
= x 
"m “El 
o 
III 
i m 
<u <= 
E p 
y Pd 
t Bd 
vow 
Ta 
El. El 
T S 
EE 
H 4 
q SI 
<< 
Eg < 
a e 
oll 
oso 


8.B Derivation of Equation (8.12) 


If we have 

1. an admissible and normal pair, (y, u), 

2. an admissible variation pair (dz, dv), and 

3. continuous first and second order partial derivatives of the integrands of 
ie j = 1,2,...,q 

then there is an infinite number of admissible pairs (y, u) 

1. whose variations, dy and du, are, respectively, equal to dz and dv; and 

2. which satisfy Equation (8.12) on p. 251, i.e., 


I;(y,u) = ôI; = 61;(y,ú;8y,0u), j=1,2,...,n+p+ qo 


where ôI; is the variation of I; at (y,ú) along (ôy, ôu), and qo is the 
number of inequality constraints satisfied as the equalities 


Ij(y,u)- 0, j=p+1,p+2,...,q0 
We prove this result in five steps. 
Step 1 Let k=n+p+qo. Then since (y, u) is normal, there exists a set of 


variation pairs 
(óyj,9u5; j—12,...,k 


such that the determinant 


ôl (6y1,0u1) ôl (dy2, duz) m ôl (dyn, Uk) 
ôl (yı, uz) ôl (dy2, duz) T 0I (Sy, uk) 

D = | | | = 0 (8.29) 
61, (9y1,9u1) 0I, (6y2,6u2) gue 0I, (dyn, OUR) 


Let there be a control 


k 
ucü- Y Bjdu; + dv 


j=1 
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for some real numbers e and 61, Pa, ..., By. Dependent on these numbers 
there are infinite such controls. From the Embedding Theorem (Hestenes, 
1966) there exists for each u a unique solution y of the differential equations 


y =8g(y,u); y(0)=y(0) + 2 B, dy (0) + €62(0) (8.30) 


Thus, there are infinitely many admissible pairs (y, u). Each y has continuous 
first and second partial derivatives with respect to 


a= Bi Bo... k € 


for the non-zero norm ||a|| < ðo for some ôo > 0. Moreover, these partial 
derivatives are piecewise continuous so that* 


dy = dz, and oy 
de a=0 OB; a=0 


T 


=0Y4+ J=1, 26. k (8.31) 


Step 2 Next, consider the equations 
F;(6,€) = Ij(y,u) — eólj(y,ujóz,óv) 20; j=1,2,...,k (8.32) 


They are obviously satisfied at a = 0 for which each I; = I;(y, i) = 0 as well 
as e = 0. Now the partial derivative of F, with respect to £5 is 


oF, _ 9h 1 [om yu -fÈ Ogi Oyi y 25 du di 
OB» OB» T OY; Of» T Bus O8» 


r- j 5 Ss t y dt = ôl (y, u; 6y2, 6u2) 
TJ Acl OY; D 
9 i= 


Qu 
Therefore, 
OF. z 
i = ôl (y, a; dy2, 6u2) = 611 (dy, dug) 
Of» a=0 
In general, 
SEE 57 i(dy;,duj;) at a=0; i,j =1,2,...,k (8.33) 
Op; 


* For example, dy1(t) = E at a = 0 is defined by the equations 


g Oy; Og Ou; Oy 
y —= = ĝyı (0 
= 3 Oy; Of D» Duy B! BBA uus (0 


which are obtained from Equations (8.30) at a = 0. 
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As a result, the Jacobian of F = Fi F> T FX i with respect to 


B= fi Ba zu Bk T at œ = 0 is the non-zero determinant D in 
Equation (8.29). From the Implicit Function Theorem (Section 9.16, p. 277), 
Equations (8.32) are valid in an open region around 8 — 0 and have solutions 


B;-—bj(e e| <ô >0; j=1,2,...,k (8.34) 
where the partial derivatives Ob;/0e are continuous, and 
b;(0)=0; j=1,2,...,k 


Step 3 Differentiating Equations (8.32) with respect to e at e = 0, and 
utilizing Equation (8.33) as well as the similarly derived equations 


I; 
Ed = 01;(y,0;0z,9v) at e=0; j=1,2,...,k 
€ 
we obtain 
91,(9y1,9u1) ôl (dy2, dug) TE ôl (Syk, OUR) dbı/de 0 
ôl (0y4,0u1) ôl (y2, duz) TE ôl (yp, uk) dba /de 
0I, (dy1, uy) SI, (dy2, u2) TEN. SI (dy y, OUR) db, /de e=0 0 
Since the determinant of the matrix in the above equation is non-zero from 
Equation (8.29), we have, in terms of b= bı ba a bk T 
> =0 at e=0 


d Oy db; = 
dy = qe y hte), el e db, de "ias — óz and 
—’ 
=0 
ôu = ôv similarly 


As a consequence, (6z, dv) is the variation pair of the family of infinite pairs 
represented by (y,u). Therefore, from Equations (8.32), which have solutions 
given by Equations (8.34), the infinitely many pairs (y, u) satisfy 


L(y, u) = €61;(y, u; dy, du); j=1,2,...,k 
Since the variation of a functional is homogeneous, 


că; (3, ü; dy, du) = ól;(y,u; eóy , edu); j— L2,...,k 
Se oM LU 


Óyo duo 
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In terms of the variations óyo and dug as indicated, we obtain 
L(y, u) = ôI; (y, ū; dyo, duo); j= Li Zinc ke 


Since óyo and dup are, after all, respective variations in y and u, we can 
rewrite the above equations as 


Ij(y,u) = ôl; (y, a; dy, du); j = 1,2,...,k (8.12) 


Step 5 Finally, we need to show that the family of infinite pairs (y, u) sat- 
isfies the inactive inequalities, i. e., the remaining strict inequalities 


Ij(y,u) <0; j=k+1,k+2,...,q (8.35) 
that are satisfied by the pair (y, ü). Now for sufficiently small e 
I;(y,u) = 1;(¥, 8) + €61;(¥, 0; ôy, ðu); j=k+1,k+2,...,q 
Also 
L(¥,a) <0; j=k+1,k+2,...,q 


Combining the last two relations for a small enough e in the interval (0, 91) 
[see Equations (8.34)], we obtain Inequalities (8.35). 


8.C Fourier Transform 


Given a function f(t), its Fourier transform, or the k-th Fourier coefficient 


is defined as 
7/2 


fik == I fte dt 
—7/2 


where w = 27/7. The function in terms of the Fourier coefficients is then 
given by the inverse Fourier transform, 


O= Y. fue 


k——oo 


which is the Fourier series. 
'The Fourier transform of the derivative of f with respect to time is 


fü) = ikw f (k) 


The last two equations can be easily verified by using the definition of the 
Fourier transform. 
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8.D Derivation of Equation (8.25) 


We need to show that each right-hand side term of Equation (8.24) on p. 254 
is equivalent to the corresponding term in Equation (8.25). We will do that 
for the first term, i.e., derive 


1 for " 
= f dy Lyyéy dt = UTG Lj,GU' 
: 
0 


'The rest of the equivalences may be similarly obtained by the reader. 
In the left-hand side of the above equation, we substitute óy from Equa- 
tion (8.21) to obtain 


T 


=] (GU)! ei! + (GU)! e*t] Lis [(GUe t a QU dt 
0 
1 / (Grim! TF Tr >» 
~ OF (GU) Lyy GU+(GU)' L,, GU dt 


0 


because the integrals of terms involving e*?'^* are zero [see Equation (8.23), 
p. 254]. Since Lyy is a symmetric matrix, the two terms of the last integrand 
are equal so that the last integral simplifies to 


= 2(GU) Lyy GUd = (GU) Ly GU = U'G L,, GU 
F 
0 


which is the first right-hand side term of Equation (8.25). 
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Exercises 


8.1 Find the necessary conditions for the minimum of 


subject to 
y = gly(t), ut), y(0)— y(7) 


where 7 is the time period that is free to change. 
8.2 Derive the stationarity condition given by [Equation (8.3), p. 242]. 


8.3 Develop a computational algorithm based on the penalty function method 
to solve Example 8.4 (p. 254) subject to the following constraints: 


T 


1 _ 
— | dr = and uz < U2 max 
m 

0 


Solve the problem for 
uj = 6 and U2.max — 10 


8.4 Apply the pi criterion to the problem in Example 8.3 (p. 241), and derive 
the condition for the vector of optimal steady state controls to be proper. 


8.5 Find the controls that satisfy the pi criterion in Exercise 8.4 and utilize 
them as initial controls to solve the optimal control problem. 


Chapter 9 


Mathematical Review 


9.1 Limit of a Function 


Consider a function f(a) defined in the vicinity of  = ro. Then the limit of 
f(x) at x = zo is a real number L approached by f(x) as x approaches xo. 
Symbolically, 

During the approach process, x and f(x) may be greater or less than the 
respective targets ro and L. The limit L may not be equal to f(xo). Also, 
the function may be not defined at zo. 


9.2 Continuity of a Function 


Consider the plot of f(x) versus z. The function is said to be continuous 
when 


1. the function values are bounded (i.e., they do not shoot to positive or 
negative infinity) and 


2. the plot, which is a curve, is not broken. 


In other words, if a function is continuous at x = xp, then we can have a 
function value f(x) as close to f(xo) as we wish by moving x near zo. Using 
the limit notation 


lim f(a) = f(vo) 


120 


where x and f(x) may be greater or less than the respective targets, zo and 
f (xo). Note that the function is defined at xo, and the limit L = f (xo). 


267 
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With the help of absolute differences, | f(a) — f(ao)| and |x — zo|, the above 
concepts are expressed more precisely as follows. 

A function f(x) is defined to be continuous at z = zo when given any e > 0 
for which |f(x) — f(xo)| < e there exists a 6 > 0 such that |x — zo| < ô. 


9.2.1 Lower and Upper Semi-Continuity 


In the definition of continuity, the inequality | f(a) — f(xo)| < e expands to 


f(zo) e< f(x) and f(x) < f(zo)- € 


A function is lower semi-continuous at xy when the left-hand inequality 


f(zxo) — e « f(x) 


is satisfied for |x — zo| < ô. 
Similarly, a function is upper semi-continuous at ro when the right-hand 
inequality 


f(x) < f(zo) + € 


is satisfied for |x — zo| < ô. 
Thus, a function f(x) is continuous when it is both lower and upper semi- 
continuous. 


9.3 Intervals and Neighborhoods 


The set of all values of x satisfying 
a<x<b, a<b 


is called an open interval and is denoted by (a,b). Note that the end points 
are not included in an open interval. 
The set of x satisfying 


a<x<b 


is called an closed interval and is denoted by [a,b]. The end points are 
included in a closed interval. Sets (a,b] and [a, b) are half- or semi-open (or 
closed). 

Open and closed intervals generalize to open and closed sets, which respec- 
tively exclude and include the boundary elements. 

A neighborhood of an element x is an open set containing zo. 
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9.4 Bounds 


A lower bound of a set is a number with the following property: 


A member of a set is either less than or equal to the lower bound. Thus, 1 
and 2 are lower bounds of [2, 3] as well as (2,3). There may be multiple lower 
bounds. Moreover, a lower bound may not be a member of the set. 


The greatest of the lower bounds of a set is called the infimum. Thus, 
2 is the infimum of [2,3] as well (2,3). Note that the first set contains the 
infimum, while the second does not. 


Similarly, a set member is either greater than or equal to the upper bound. 
The least of the upper bounds of a set is called the supremum. 


9.5 Order of Magnitude 


The order of magnitude of a real number is how many tens the number 
has. Thus, the order of magnitude of x is the integer part of the log; x. 


An n orders of magnitude difference between two real numbers is a difference 
by a factor of 10”. 


9.5.1 Big-O Notation 


Consider a function f(x) as x tends to some value zo. If the absolute value of 
the function is bounded by some positive constant c multiplied by the absolute 
value of x when «x is sufficiently close to zo, then we can write 


lim |f(z)| < ele], |r—zo| <6 >0 
z—2X0 


'The shorthand for the above expression is 


lim f(x) = O(x) 


T1— 20 


where O(x) represents the bound clx| as x tends to xo. 


270 Optimal Control for Chemical Engineers 


9.6 Taylor Series and Remainder 


The estimate of a function f(x) at a distance h from the reference point z = to 
is given by the n-th order Taylor series or expansion 
df h? ud h? df h” d^f 
h) = h — LER EA sea 
Pao +h) = Hizo) + dz a si 2! dz? Lo. 3! dr? aa n! dg” 
hn df 
+ = === 
(n+ 1)! dz", 


m —= 
remainder 


where C lies in the interval (zo, zo +h). In the above equation, the last term 
is the remainder, which represents the sum of the remaining infinite number 
of terms 

Arti darti f pn+2 dtr f 


(RI det... ey dee , bie 


For a fractional value of h, the remainder becomes negligible as n is increased. 
For sufficiently large n or small h, the remainder is therefore discarded. It is 
of course assumed that the function is differentiable at least n times. 


9.7 Autonomous Differential Equations 


An autonomous differential equation does not carry the independent vari- 
able explicitly, e. g., 
duo 
dt 
On the other hand, the following is a non-autonomous differential equa- 
tion: 


= You 


9.7.1 Non-Autonomous to Autonomous Transformation 


Any non-autonomous differential equation can be transformed into a set of 
autonomous differential equations by introducing additional state variables. 
For example, with the introduction of the new state variable 


yo —t 
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the last non-autonomous differential equation is transformed into the following 
set of autonomous differential equations: 


dy2 


= Y1Y2 — uys and ra =1 


dy 
dt 


9.8 Differential 


Let f be a function of the independent variable z. Then the differential of f 
at x = xg is defined as the change df in f corresponding to the change h in 
x from zo meeting the following requirements: 


1. The differential df is closer to the actual function change 
f (zo +h) — f(x) 
than h, which is a real number close to zero. 


2. The differential df is a linear and continuous function of h at x. 


Remarks 


'The motivation for the above requirements is simplification. With negligible 
error, we would like to have a simple function d f represent the function change 
corresponding to a variable change. 

Alternatively, we would like to compute the new function value fi = f(xo-- 
h) in a simpler way* using 


fi = f(xo) + df(zo; h) 


for small enough values of À that obviate any errors. The two arguments of 
df denote the dependency on the change h at a given zo. 

Putting it all together, the differential df of f at xo is a linear and contin- 
uous function of the variable change and satisfies 


f(xo +h) = f(xo) + df (xo; h) + elh) 


where the error e vanishes faster than h. So if we keep on reducing h, after 
a while when h takes a certain value ho, the error e would disappear and df 
would be an accurate representation of the function change in the interval 
[Zo, XO + hol. 


* As opposed to evaluating fi = f(xo + h) all over again from the functional relationship. 
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9.9 Derivative 


The definition of the derivative follows from the differential. We multiply and 
divide df(x0;h) by h in the last equation to obtain 


à: d f (zo; h) 


— ë — 


f(zo+h) = f(xo) h+ e(h) 


derivative 
where the derivative is the coefficient of h and is defined as 


ga Eh — AF] _ uo flo +h) = Feo) 


h=>0 h dx h=>0 h 
TO 


9.9.1 Directional Derivative 


The directional derivative of a function f(y) at y = yo along a unit vector v 
is defined as 
(yo + av) — f(yo) 


Vyf(yo) = lim 
a—0 Q 


Thus, V, f(yo) quantifies the rate of change in f from y = yo along v. 
If f(y) is differentiable at yo, then V, f(yo) exists along any unit vector. 
Moreover, if y = yo + av, then f = f[y(a)], and 


Vvf(yo) = tin ID FO = Si 


a—0 Q da a 
Of dyi ". ð 
DEZ MEE 
ici CY PU. <p ici Yi yo 
where i denotes the vector i evaluated at yo. 
Example 
The directional derivative of 
aa 
y ty 
at yo = v3 -v2 T along the unit vector v = y/3/5 2/5 Dis 


1 
Vv f(yo) = yi Ya Ui Va — 5 
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9.10 Leibniz Integral Rule 


Consider the following definite integral 
b 
I= J f da 


Then the derivative of J with respect to its upper limit is given by 


b+Ab b 
f fdz- f fae 
dI NEP I(b + Ab) — I(b) NET a a 
db — A Ab B Nin Ab 
b--Ab 
J f da 
Ls Wess ci _ f(bAb — 
A: Ap COM 


Similarly, the derivative of I with respect to its lower limit is given by 


Z = la) 


9.11 Newton-Raphson Method 


This is a numerical method to find the root of the function, i.e., the value of 
the independent variable that makes the dependent function zero. Using the 
first order Taylor expansion, the function value at z;,1 can be expressed as 


Hui) E f(x) + f (21) (£iz1 — 2i) 


When 2;+1 is sufficiently close to the root, 


0S fim files — zi) (9.2) 
Algorithm 
The above equation provides the following algorithm: 
f(x) ; 
il = Ti — ; = 1,2 
Lil ST EN 1 


which is repeatedly applied to improve an initial guess xo for the root. 
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Quadratic Convergence 


Sufficiently close to the root x,, the second order Taylor expansion gives 


where ¢ lies in the interval (x;,x,). From the above equation, we subtract 
Equation (9.2) to get 


f^ (C) 


2 
Ly — X441 = — (Tr — Ti 
Gata a 
new error old error 


Thus, the new error is proportional to the square of the old error. If the old 
error is 0.1, then the new error would be of the order 0.01. The next error 
would be of the order 107*, and so on iteratively. The value x;+1 is said to 
converge quadratically to the root zr. 


Extension to Multivariable Functions 


The Newton-Raphson algorithm for multivariable functions is 
Xizi = sepale Peay 1250452. 


where J; is the Jacobian 


ah ah of 
DU v On2 x, ae Ov. x, 
oh oh op 
dm x, Dr 5 dn x, 
Of tu Os 
Oz x, Ora x, mE tn x, 


evaluated at x = X;. 
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9.12 Composite Simpson's 1/3 Rule 


The basic Simpson's 1/3 Rule rule provides the value of the integral 


b 
T= [fod = TS pay rag E 


+ f(b) 


by approximating the integrand f(x) with the quadratic function that inter- 
polates the function values f(a), f[(a + b)/2], and f(b). 

The value of I is improved by dividing the integration interval into a num- 
ber of segments of equal length, applying the Simpson's 1/3 Rule to each 
segment, and summing up the resulting values. This procedure finally yields 
the following composite Simpson's 1/3 Rule 


XN N N 
r= [feas = $ fG)4 E se)+2 Y; fis Flew) 
¿=1,3,5,... ¿=2,4,6,... 


where h is the distance between any two successive z;s forming a subinterval 
[z;, z;41] where ¿=0,1,..., (N — 1). 

Note that each segment in the integration interval [ro, zw] has two subin- 
tervals. Thus, the number of subintervals, N, is even and the number of x;s 
being (N + 1) is odd. 


9.13 Fundamental Theorem of Calculus 


If a function f(x) is real and continuous in a closed interval [a,b] such that 
the primitive integral or the antiderivative 


F(a) = frs 


is defined, then F(x) is continuous in [a,b], and differentiable in (a,b). The 
derivative of F with respect to x in (a,b) is given by 
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Moreover, if in the closed interval [a,b] the function f is real, but not 
necessarily continuous, and has the antiderivative F, then 


r$) - r9) « | fase [Lao 


9.14 Mean Value Theorem 


If f(x) is a continuous function in the z-interval [a,b], then there is a real 
number c in the interval such that 


b 
(afl = f faz 


9.14.1 For Derivatives 


Let f(x) be continuous in the z-interval [a,b] with the derivative dy/dx de- 
fined in the open interval (a,b). Then there is point zo in (a,b) such that 


df 
dz 


To 


f(b) — f(a) = (b — a) 


This result is known as the law of the mean or the Mean Value Theorem for 
derivatives. 


9.14.2 For Integrals 


Let f(x) be continuous in the x-interval [a,b]. Then there is point zo in [a,b] 
such that 


b 
J (as - 6- fi) 


'This result is known as the Mean Value Theorem for integrals. 


9.15 Intermediate Value Theorem 


Let f(x) be continuous in the z-interval [a,b] and f(a) = f(b). Then according 
to this theorem, f(x) assumes each value between f(a) and f (b) as z changes 
from a to b. 
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9.16 Implicit Function Theorem 


Consider the set of functions 


Pit) = €] 
AU En) = C2 
Fnl[%1,%2,..., Tn) — Cm 


where m < n. Then according to this theorem, provided that 


e the vector of functions 
F 


f= f(x) fa) o Sa) 
is differentiable near a, 
e the derivatives of f with respect to x are continuous at a, and 
e the Jacobian determinant fx is not zero at x = a. 


we can obtain the set of solutions 


LI = gı(Em+1; Um+2> ee silly) 
T2 = ga mais Tm+2)00-> En) 
Tm = Im[Em+1> Tm+2)00.> Zn) 
near 
= 
x=a= ay Q2 2 An 
where g;s are differentiable functions of 41, 25442, ..., and Lp. 


9.17 Bolzano—Weierstrass Theorem 


According to this theorem, there is at least one point of accumulation (or limit 
point) in a bounded set having an infinite number of elements. Alternatively, 
each bounded sequence in the set has a subsequence that converges to a point 
in the set. 

The point of accumulation and subsequence are described as follows. 
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Point of Accumulation 


Not necessarily in a set, its point of accumulation has in its each neighborhood, 
at least one non-identical point from the set. A closed set contains all of its 
accumulation points. 


Subsequence 


Given a sequence 
Ti, T2, U3, Ta, U5, -.. 


its subsequence is, e. g., 
X1; T3, T4, T6; T8, "wor 


which is contained in the original sequence. The order of the elements is 
preserved in the subsequence. 


9.18 Weierstrass Theorem 


According to this theorem, a function f(x) in a closed and bounded domain 
must attain minimum and maximum values. This theorem is also known as 
the Extreme Value Theorem. 


9.19 Linear or Vector Space 


A linear or vector space is a set of elements known as vectors for which the 
following two operations: 


1. addition of vectors and 
2. multiplication of a vector by real numbers 
are defined. These operations satisfy the following rules: 


1. For any two vectors x and y, 
i. the sum (x + y) is a vector 
ii. 2+y=y+ 
iii. (1+y)+2=x+(y+2) for any vector z 
2. For any real number a, 


i. the product az is a vector where x is any vector 
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ii. a(x +y) = ox + ay for any two vectors x and y 


3. For any two real numbers a and f and any vector x 
i. (a + B)z = ox + Ba 
ii. a(Bx) = (a)x 

4. 'The space contains 


i. the zero vector, 0, such that x +0 = x 
ii. the unit vector, 1, such that la =a 
iii. the negative of x, denoted by —z, such that x + (—x) =0 


for any vector z. 


Examples 
Examples of vector spaces are 


1. The set of real numbers for which the usual addition and multiplication 
are defined. 


2. The set of elements, each made of n components in an order for which 
vector addition and scalar multiplication are defined. For example, an 
element x is given by 


Tı 

T2 
X = 

Tn 


3. The set of real-valued functions defined on a fixed interval of the inde- 
pendent variable. For example, if any two such functions or vectors are 
p(x) and q(x) for x in [a,b], then the respective operations of addition 
and multiplication (by real number a) result in vectors 


r(x) =p(x)+q(1) and s(x) = ap(z) 
which are also vectors. 


It is easy to verify that the above examples obey the rules of a vector space. 


9.20 Direction of a Vector 


The direction of an n-component vector 


v= v V2 22s Ui i Un, 
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is characterized by the set of n direction ratios 
(di, da, ..., di, ..., dn} 


where 


Two vectors along the same direction have an identical set of direction ratios 
and vice versa. 


9.21 Parallelogram Identity 


For any two vectors a and b and a scalar a 


(a; + aby? 


M: 


Il 
= 


la + abl? = 


(a? + o?02 + 2aa;b;) 


II 


e 
Il 
= 


= lla? + 0? ||bl|? + 2aa" b 


9.22 Triangle Inequality for Integrals 


Given a function f(x) 
-|f()| € f(z) € |f()l 


Integrating each term in the above relation, we obtain 


- fires f pear fiaa 


which is equivalent to 
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9.23 Cauchy-Schwarz Inequality 


The inequality is 
a'b < lla]! [bl] 


where a and b are two n-dimensional vectors. 


For n = 2, let 

a b 

a= d and b= i 

ag ba 

Then 
(aj + a3) (07 +b3) — (aibi + a2b2) = (aiba — ab)” 
N oe ~N — Se a A ES 
all? bl? (a! b)? >0 


Therefore, a! b < ||a|]]|b||, which can be easily generalized for higher dimen- 
sions. 


9.24 Operator Inequality 


The inequality is 
| Ab|| € [ATI 


where A is an n x m operator (matrix), and b is an m-dimensional vector. 
For n — m — 2, let 


Then 
Ab= a1101 + a1202 
a21b1 + a22b2 
and 
[AD]? = (a11b1 + a1202)? + (aibi + a22b2)? 
————À == Se p 
ti to 
Since 
(aj, + a3) (bi + b2) — ta = (a11b2 — a12b1)? > 0 
we get 


tı € (ad + a2)(01 + 03) 
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Similarly, 
to < (a3; + a39)(01 + 63) 


Adding the last two inequalities, we get 


titte < (at, + a + a5; + a32) (bf + 85) 
N = —— ee’ A, — 


|| Ab? ¡ENS bp 
The above result in terms of A and b is 
|Ab|) < ||Al||b]| 


which can be easily generalized for higher dimensions. 


9.25 Conditional Statement 


A conditional statement is a conjunction of a condition and the outcome 
generated by the condition. Consider the following statement: 


If you drop the glass then it will crack. 
N ea 


N Y = — 
condition (A) — outcome (B) 


which can be symbolically represented as 
A — B 


As the arrow indicates, the flow of events is from A to B but not necessarily 
vice versa. In other words, B —> A is not necessarily true. For example, 
if the glass cracked, then it was not because you dropped it but you poured 
boiling water into it. 

Given a conditional statement A —> B, the satisfaction of A causes B. Con- 
dition A is known as the sufficient condition. Its satisfaction is sufficient 
to cause the outcome B, which in turn is called the necessary condition. 
Outcome B is necessary for condition A to have been true. Put differently, if 
B did not happen, then neither did A. Using — for negation, 


=B — A 


Thus, if the glass did not crack, then you did not drop it for sure. 

The conditional 4B —> —A is called the contrapositive of the original 
conditional A —> B. The two conditionals are equivalent. One is the contra- 
positive of the other. 
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9.26 Fundamental Matrix 


Consider the homogeneous linear differential equation 


in a time interval where x = x(t) is an n-dimensional vector, and A isannxn 
matrix of constants. Then n-linearly independent solutions of the differential 
equation exist. Let us denote the solutions by column vectors: 


xi(t), Xa(t), ..., Xn(t) 


Then the fundamental matrix is the collection 


The general solution of the differential equation is 
x= cxi(t) = W(t)c 
i=1 


where c is a vector of some constants. If the initial condition at t = tọ is 
x(to), then from the above equation, 


c= w | (to)x(to) 
and the solution in terms of the fundamental matrix is 


x = U(t)W~*(to)x(to) 
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inactive, 110, 111, 250, 251 
infimum, 138, 142, 144, 269 
integration by parts, 49 
Intermediate Value Theorem, 53, 
276 
interval 
closed, 268 
open, 268 
inverse Fourier transform, 253, 
264 
Inverse Function Theorem, 90, 92, 
95, 115, 117 


Jacobian, xvii, 191, 240, 243, 263, 
274 
determinant, 90-92, 277 
John Multiplier Theorem, 109 
generalized, 113 
preconditions, 111, 113 
jump discontinuity, 76, 77, 127, 
177 


Lagrange Multiplier Rule, 45, 46, 

48, 59, 61, 67, 93, 94, 99, 
100, 102, 106, 110, 115, 
153, 154, 169, 179, 230, 
250 

for several equalities and 
functions, 96 

for several equality 
constraints, 96 

for several states and 
controls, 103 

in optimal control problems, 
99 

preconditions, 94, 96, 102, 
104, 178 

several constraints and 
functions, 98 

Lagrange Multiplier Theorem, 59, 

88 

proof, 90 
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Lagrange multipliers, 59 
motivation, 87 
relation to objective 
functional, 107 
role, 88 
Lagrangian, 163, 165-167, 169, 
170, 210, 214, 237, 249, 
255 
Leibniz Integral Rule, 154, 273 
linear spaces, see vector spaces 
local optimum, 73, 73, 76, 185 


matrix 
Jacobian, xvii, 191, 240, 243, 
263, 274 
maximum 
necessary condition, 58 
relation to minimum, 185 
Mean Value Theorem, 276 
for derivatives, 42, 119, 140, 
141, 276 
for integrals, 62, 100, 276 
minimum 
necessary condition, 58 
relation to maximum, 185 
multiple integral problems 
gas diffusion in polymer, 178 


necessary conditions, 52, 57-61, 
64-71, 73-76, 81, 83, 84, 
87-90, 93-97, 99, 101, 
102, 107, 109-114, 121, 
123, 125, 126, 128, 153, 
156-162, 164-167, 
170-172, 174, 176-179, 
181-183, 186, 189, 194, 
202, 207, 210, 217, 221, 
224, 226, 230, 233, 
236-238, 242, 250, 260, 
266, 282 

neighborhood, 268 

Newton-Raphson method, 191, 
223-225, 240, 273 

multivariable, 274 
norm, 26 
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properties, 26 
normal 
control, 251 
pair, 251 
normality condition, 89, 251 


objective functional, 3 

autonomous, 126 
operator inequality, 118, 281 
optimal control, 1 


final state on 
hypersurfaces, 162 
fixed final state, 161 
free final state, 161, 199 
free final time 
final state on 
hypersurfaces, 158, 201 
free final state, 153, 185, 
207 
heat exchanger, 8 


advantage, 4 
laws, 20 
subject, 1, 4 


optimal control problems 


an outline of solution, 71 
batch distillation, 5 
with integral equality 
constraint, 170 
with integral inequality 
constraint, 172 
batch reactor, 157, 159, 203, 
inequality constrained, 167 
blood flow and metabolism, 
17 
chemotherapy 
with interior point equality 
constraints, 174 
chemotherapy for cancer 
treatment, 15 
classification, 19 
control inequality constraints, 
113 
CSTR, 69, 104, 225 
equality constrained, 164 
inequality constrained, 166 
with algebraic constraint, 
107 
diffusivity determination, 9 
dispersion determination in 
Vapex, 13 
end point inequality 
constraint, 111 
fed-batch reactor, 192, 199 
fixed final time, 223 


local optima, 73 
medicinal drug release, 16 
nuclear reactor, 12 
periodic control of 
polymerization, 11 
plug flow reactor, 6 
simplest, 58 
general, 67 
with algebraic equality 
constraints, 105, 163, 209 
with algebraic inequality 
constraints, 166, 217 
with integral equality 
constraints, 168, 214 
with integral inequality 
constraints, 171, 221 
with interior point 
constraints, 172 
optimal periodic control, 235 
batch reactor, 236 
CSTR, 237, 241 
necessary conditions, 236 
shooting Newton-Raphson 
algorithm, 240 
solution methods, 239 
shooting Newton-Raphson, 
239 
successive substitution, 239 
optimal steady state, 235, 248, 
250, 252 
necessary conditions, 260 
optimization, 4 
order of magnitude, 269 


Parallelogram Identity, 151 
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parallelogram identity, 280 
penalty function, 202, 210, 215, 
217, 219, 221 
penalty function method, 185, 
201, 201, 202, 205, 207, 
214, 217, 219, 233 
algorithm, 203, 266 
weighting matrix, 202-204, 
210, 212, 217 
perturbation 
effect of final state, 134 
pulse, 130, 131 
temporal, 131, 134 
pi criterion, 248 
assumptions, 250 
periodic CSTR, 254 
with control constraints, 256 
periodic CSTR, 259 
piecewise continuous controls, 76, 
127 
Pontryagin’s maximum principle, 
123, 126, 136 
Pontryagin’s minimum principle, 
123-125, 128, 129, 156, 
178 
application, 123 
assumptions, 128 
class of controls, 126 
derivation, 129 
notation, 127 
outline, 129 
proper 
control, 250 
pair, 250 


second variation, 50, 50, 51 
homogeneity, 50 
shooting Newton—Raphson 
method, 223, 223, 225, 
226, 239, 242, 243 
algorithm, 225 
Simpson’s 1/3 Rule, 275 
composite, 187, 189, 195, 211, 
218, 244, 275 
state equation, 64 
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state equations, 58, 59, 64-73, 83, 
106, 107, 111, 121, 128, 
131, 133, 134, 174, 175, 
182, 186, 187, 189, 190, 
192, 193, 195, 202, 
223-226, 239-244, 250 
autonomous, 126 
stationarity conditions, 61, 64-69, 
71, 126, 182, 223, 224, 
226, 242, 266 
steepest descent, 190, 191, 229 
subsequence, 144, 152, 277, 278 
sufficient conditions, 52, 57, 73, 
74, 252, 255, 259, 282 
Mangasarian, 74, 81 
pi criterion, 248, 257 
properness of optimal steady 
state, 259, 260 
weak, 75, 76 
supporting hyperplane, 135, 149 
supremum, 269 


Taylor expansion 
n-order, 270 
first order, 108, 125, 133, 134, 
155, 273 
remainder, 270 
second order, 51, 52, 230, 274 
Taylor series, see see Taylor 
expansion 
trajectory optimization, 4 
triangle inequality, 26, 116, 150 
for integrals, 43, 280 
two point boundary value 
problem, 223, 224, 226, 
239 


variable 
adjoint, 59 
costate, 59 
independent, 58 
state, 58 

variation 
homogeneity, 36, 57 
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variation of functional, 23, 36, 45, derivatives, 41, 44, 72, 76, 
47, 48, 50, 52, 57, 62 182, 186, 187 
of integral objective, 40, 49 pair, 252, 253, 258 
variation of integral vector spaces, 23, 25, 26, 278 
equivalence to Fréchet 
differential, 42 Weierstrass Theorem, 140, 150, 
equivalence to Gáteaux 278 
differential, 43 Weierstrass-Erdmann corner 


variational conditions, 78, 178 


